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PREFACE 


This  description  of  methods,  procedures  and  instruments  for  measuring  job  performance  by  the 
productive  capacity  of  enlisted  personnel  is  part  of  an  on-going  Air  Force  research  program  to  develop 
the  technology  necessary  to  base  selection,  classification  and  personnel  management  policies  on 
empirically-derived  performance  data.  The  effort  was  undertaken  by  the  Manpower  and  Personnel 
Research  Division,  Human  Resources  Directorate,  Air  Force  Armstrong  Laboratory  as  part  of  Work  Unit 
29220204,  "Linking  Job  Performance  and  Enlistment  Standards."  The  work  is  also  in  response  to  the 
following  Manpower,  Personnel  and  Training  Needs  (MPTNs): 

MPTN  89-05MP  -  Measurement  of  Productivity 

MPTN  89-1  IMP  -  Enlisted  Selection  and  Classification  Technology 

MPTN  89-13MP  -  Job  Performance  Measurement 

The  project  was  conducted  from  September  1990  to  September  1991  with  the  following  overall 
objectives: 

To  develop  procedures  and  instruments  for  measuring  the  productive  capacity  of  enlisted  personnel; 
and 

To  collect  productive  capacity  data  for  four  enlisted  Air  Force  Specialties  (AFSs),  one  from  each 
aptitude  area  (Mechanical,  Administrative,  General  and  Electronic). 

This  report  addresses  only  the  methodological  issues  of  measuring  productive  capacity.  Descriptions 
and  analyses  of  the  data  collected  during  this  effort  will  be  the  subject  of  subsequent  Armstrong 
Laboratory  reports. 

The  authors  wish  to  express  their  appreciation  to  the  many  men  and  women  of  Strategic  Air  Command 
(SAC),  Tactical  Air  Command  (TAQ,  the  Military  Airlift  Command  (MAO  and  Air  Force  Systems 
Command  (AFSQ  whose  willing  participation  as  test  subjects  made  this  effort  possible.  In  particular,  we 
wish  to  thank  our  points  of  contact  who,  in  spite  of  DESERT  SHIELD/DESERT  STORM  deployments 
to  the  Middle  East,  arranged  for  the  facilities  and  test  subjects  required  to  accomplish  this  project  We 


also  wish  to  acknowledge  the  technical  guidance  and  assistance  provided  Dr.  William  B  Alley  and 
Dr.  Lonnie  D.  Valentine  of  the  Manpower  and  Personnel  Research  Division,  and  Dr.  Mark  S.  Teachout 
of  the  Technical  Training  Research  Division,  Human  Resources  Directorate,  Air  Force  Armstrong 
Laboratory.  Data  collection  assistance  was  also  provided  by  Capt  Rudolph  A.  Smith  and  lLt  Mark  R. 
Miller,  both  from  the  Manpower  and  Personnel  Research  Division. 
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SUMMARY 

This  document  describes  die  development  of  procedures  and  instruments  for  measuring  the  relative 
productive  capacity  (PC)  of  Air  Force  enlisted  personnel.  In  the  context  of  this  study,  an  individual 
airman’s  PC.  on  to  a  particular  task  in  his  or  her  Air  Force  Specialty  (AFS),  is  defined  as  the  fastest 
possible  time  in  which  the  task  could  be  performed  divided  by  the  time  it  would  take  the  airman  to 
perform  the  task,  as  estimated  by  the  airman’s  supervisor.  The  airman’s  overall  PC  in  the  AFS  is 
calculated  by  averaging  the  PC  values  across  a  sample  of  tasks. 

PC  is  of  interest  to  Air  Force  personnel  planners  and  managers  as  a  potential  measure  of  relative  work 
output,  which  could  be  used  to  develop  more  cost-effective  selection,  classification  and  retention  policies. 
While  the  marginal  costs  of  increasing  aptitude  and  experience  levels  in  the  enlisted  force  can  be 
estimated,  the  marginal  improvement  in  work  output  resulting  from  these  higher  aptitude  and  experience 
levels  is  currently  unknown.  As  a  result,  meaningful  cost-benefit  analyses  are  precluded.  However,  if  a 
practical  method  for  measuring  PC  can  be  developed,  and  if  PC  can  be  linked  to  aptitude  and  experience, 
the  Air  Force  will  have  a  valuable  new  tool  for  more  efficiently  managing  its  people. 

To  evaluate  die  feasibility  of  using  supervisor  estimates  of  task  performance  times  to  determine 
individual  PC  values,  about  320  supervisors  in  four  AFSs  (Aircrew  Life  Support,  Aerospace  Ground 
Equipment,  Avionic  Communication  and  Navigation  Systems,  and  Personnel)  were  asked  to  estimate 
performance  times  for  approximately  680  of  their  subordinates.  The  supervisors  were  predominately  Staff 
and  Technical  Sergeants  with  around  12  years  of  service  while  their  subordinates  were  3-  and  5-  skill-level 
airmen  with  less  than  6  years  of  service.  The  data  were  collected  at  ten  Air  Force  bases  in  four  Major 
Commands:  Tactical  Air  Command  (TAC),  Strategic  Air  Command  (SAC),  Military  Airlift  Command 
(MAC),  and  Air  Force  Systems  Command  (AFSC). 

To  validate  the  supervisor  time  estimates,  actual  performance  times  were  measured  on  a  sub-sample 
of  about  240  airmen.  For  construct  validation,  airmen  in  three  of  the  four  AFSs  were  given  a  job 
knowledge  test  which  had  been  developed  as  part  of  an  earlier  Air  Force  job  performance  measurement 
research  project  All  airmen  were  also  administered  an  interest  inventory  and  a  motivation  scale  (also 
products  of  earlier  Air  Force  research  efforts)  as  possible  predictors  of  PC. 
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While  the  procedures  and  instruments  used  in  this  study  are  feasible  for  large-scale  application, 
numerous  administrative  and  logistical  problems  were  encountered.  Collecting  PC  data  at  the  task  level 
is  a  complex,  labor  intensive  and  time  consuming  process.  A  comprehensive  and  detailed  data  base  was 
created  which  can  support  validation  and  other  analytical  studies— in  addition  to  these  analyses,  future 
research  should  consider  alternative  definitions  of  PC  that  require  less  detailed  data,  and  data  collection 
procedures  that  are  less  time  consuming  and  costly. 
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MEASUREMENT  OF  PRODUCTIVE  CAPACITY: 

A  METHODOLOGY  FOR  AIR  FORCE  ENLISTED  SPECIALTIES 

L  INTRODUCTION 

The  Problem 


The  Air  Force  has  long  maintained  that  people  are  its  most  important  resource.  With  the  advent  of 
the  all-volunteer  force  and  the  need  to  compete  in  the  labor  market  for  talented  people  to  operate  and 
maintain  its  sophisticated  weapons  and  systems,  people  have  also  become  the  Air  Force’s  most  expensive 
resource.  The  FY  91  Air  Force  budget  for  active  duty  military  personnel  exceeds  $20  billion-nearly 
$40,000  per  person.  Recruiting  and  training  costs,  pay  and  allowances  and  especially  retirement  and 
medical  costs  have  all  increased  dramatically  in  the  past  two  decades. 

With  labor  costs  of  this  magnitude,  it  is  imperative  that  the  Air  Force  get  the  most  productivity  for  its 
personnel  dollars --the  problem  is  how  to  measure  productivity  in  a  military  environment.  Unlike  the 
private  sector,  where  an  individual’s  contribution  to  corporate  profits  can  often  be  quantified,  the  Air  Force 
has  no  readily  available  output  measure  of  an  individual’s  worth.  Consequently,  decisions  about  who  to 
select,  in  what  jobs  to  place  them,  and  how  long  to  keep  diem  are  sometimes  subject  to  debate.  For 
example,  a  recent  Air  Force  Times  article  (West,  1991)  cited  a  January  report  by  die  General  Accounting 
Office  (GAO)  as  criticizing  the  Air  Force  for  keeping  more  career  airmen1  than  it  needed  during  die  FY 
86-89  period.  The  GAO  estimated  that  the  excess  experience  cost  the  taxpayers  $41.4  million  in  1989. 
According  to  the  article,  the  Department  of  Defense  responded  to  the  GAO  study  by  citing  greater 
readiness,  a  more  experienced  and  productive  force,  and  savings  in  accessions  and  training  costs  as 
justification  for  exceeding  the  planned  career  force  objective.  What  seems  to  be  missing  from  this  debate 
is  a  quantifiable  output  measure  by  which  military  personnel  policies  and  management  actions  can  be 
evaluated  so  that  the  costs  and  benefits  of  such  policies  and  actions  can  be  assessed  objectively. 

In  the  absence  of  empirical  relationships  among  individual  characteristics  and  productivity,  the  Air 
Force  manages  its  people  by  setting  minimum  requirements  on  factors  that  are  thought  to  be  related  to 


1  Defined  as  those  with  more  than  4  years  of  service. 
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productivity2.  Personnel  managers  then  select,  classify,  train,  assign,  promote  and  retain  or  separate 
airmen  from  the  pool  who  meet  the  minimum  standards --with  individual  preference  playing  a  significant 
rale  in  the  process.  Statistics  on  the  various  input  factors  are  maintained  and  periodically  reported  as 
surrogates  for  the  missing  output  measures.  For  example,  the  proportion  of  new  recruits  with  high  school 
diplomas  and  those  scoring  in  the  upper  half  of  the  Armed  Forces  Qualification  Test  (an  Armed  Service 
Vocational  Aptitude  Battery  (ASVAB)  composite)  are  routinely  reported  to  the  President  and  Congress 
as  indirect  indicators  of  military  personnel  capability3. 

To  reduce  some  of  the  uncertainty  in  military  personnel  management,  the  Air  Force  has  conducted 
scientific  studies  to  independently  validate  some  of  the  factors  on  which  force  management  decisions  are 
based  against  relevant  performance  indicator..  For  example,  a  strong  empirical  relationship  between 
education  level  and  first-term  attrition  (dropping  out  of  the  Air  Force  prior  to  completion  of  an  initial 
service  obligation)  has  been  demonstrated  in  numerous  studies-high  school  drop-outs  are  also  likely  to 
quit  the  service  (see,  for  example,  Laurence,  1987).  The  ASVAB  has  been  shown  to  be  a  reasonably  good 
predictor  of  success  in  technical  training  (more  or  less  so  depending  on  the  training  course)  (Ree  &  Earles, 
1991;  Wilboum,  Valentine,  &  Ree,  1984).  Also,  job  interest,  as  measured  by  the  Vocational  Interest  for 
Career  Enhancement  (VOICE)  instrument,  correlates  well  with  subsequent  job  satisfaction  (Alley, 
Wilboum,  &  Betberich,  1976).  While  die  findings  of  these  studies  lend  support  to  military  force 
management  practices,  they  address  the  key  issue  of  productivity  only  tangentially. 

Definition  of  Productive  Capacity 

Mote  recently,  the  Human  Resources  Directorate  of  the  Air  Force  Armstrong  Laboratory  has  conducted 
a  series  of  studies  aimed  more  directly  at  establishing  relationships  between  individual  characteristics  and 


2  For  example.  Air  Force  Regulation  (AFR)  39-1,  Enlisted  Personnel,  sets  minimum  Armed 
Services  Vocational  Aptitude  Battery  (ASVAB)  composite  cutoff  scores  for  entry  into  each  enlisted 
specialty. 

3  For  example,  see  page  40  of  the  January  1991  Annual  Report  to  the  President  and  the  Congress. 
by  Defense  Secretary  Dick  Cheney,  U.S.  Government  Printing  Office,  WashingtoaD.C 
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actual  performance  on  the  job.  One  set  of  studies,  conducted  as  part  of  the  broader  Department  of 
Defense  (DoD)  Job  Performance  Measurement  System  (JPMS)  project,  developed  techniques  to  measure 
actual  job  performance  in  a  military  setting  (Hedge  &  Teachout,  1986).  Numerous  techniques  were 
investigated,  including  hands-on  performance  testing,  interviews,  written  tests,  and  supervisor  and  peer 
ratings --all  aimed  at  determining  how  well  an  individual  performs  a  particular  set  of  tasks.  ASVAB  scores 
were  then  correlated  with  these  criterion  measures  as  a  means  of  demonstrating  test  validity  for  job 
performance  as  well  as  training  success.  For  the  limited  sample  of  Air  Force  Specialties  (AFSs)  studied, 
there  appears  to  be  a  reasonably  good  relationship  between  aptitude  and  job  performance. 

However,  the  JPMS-developed  performance  metrics  (generally  percent  correct  on  a  performance  test) 
are  not  ideally  suited  for  force  structure  planning,  modeling  or  analysis  purposes.  Being  able  to  predict 
an  individual’s  performance  test  score  from  their  ASVAB  scores  and  experience  level  still  leaves  the 
policy  analyst  with  die  problem  of  deciding  what  performance  level  is  required.  The  performance  metrics 
also  do  not  facilitate  decision  making  about  trade-offs  among  alternative  manpower  mixes,  an  important 
consideration  in  developing  efficient  force  management  policies. 


To  correct  this  deficiency  the  Air  Force  is  exploring  an  alternative  performance  metric  called 
productive  capacity  (PC)  that  focuses  more  on  individual  differences  in  die  amount  of  work  performed 
rather  than  just  die  quality  of  the  work  performed.  The  technique  involves  measuring  an  individual’s 
performance  times  across  a  representative  sample  of  tasks  in  their  AFS  and  then  comparing  those  times 
with  estimates  of  the  fastest  possible  performance  times  for  the  same  tasks.  The  performance  metric  thus 
generated  provides  an  estimate  of  the  amount  of  work  the  individual  could  produce  relative  to  the  most 
productive  person  in  that  AFS. 


The  productive  capacity  of  an  individual  airman  in  a  particular  AFS  can  be  expressed  mathematically 


as: 


where, 

PC,  =  Productive  capacity  of  airman  a 

T>  =  Fastest  possible  performance  time  for  task  i  (i  =  1  to  n) 

T„  =  Airman  a’s  actual  performance  time  on  task  i 
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PC  values  theoretically  range  from  0  (no  productive  capacity)  to  1  (maximum  possible  productive 
capacity);  for  example,  a  PC  value  of  .75  for  a  particular  airman  in  a  particular  AFS  indicates  that  he  or 
she  could  produce  75  percent  of  the  work  produced  by  the  fastest  possible  airman  in  that  AFS.  While  lJC 
is  not  an  absolute  measure  of  output,  it  does  provide  a  relative  measure  amenable  to  comparisons  of 
alternative  personnel  policies  and  management  actions-a  potentially  useful  personnel  policy  analysis  tool. 

Background/Literature 

While  documented  time  studies  date  back  as  far  as  1760,  Frederick  W.  Taylor  is  generally  regarded 
as  the  father  of  modem  time  study  in  this  country  (Niebel,  1976).  Taylor,  a  mechanical  engineer,  began 
his  work  in  1881  to  introduce  scientific  discipline  into  the  management  process.  He  advocated  the 
division  of  work  into  tasks  and  elements  and  the  collection  of  task  performance  time  data  to  establish 
performance  standards  and  rewards  systems.  His  work  was  die  precursor  to  modem  industrial  engineering, 
and  is  the  basis  for  die  Air  Force’s  manpower  requirements  determination  process.  However,  industrial 
engineering  time  studies  were  not  concerned  with  determining  what  worker  characteristics  account  for 
individual  differences  in  task  performance  dmes-issues  of  vital  interest  to  die  Air  Force  and  the 
motivation  behind  the  present  study.  The  literature  is  sparse  on  the  use  of  task  performance  times  as  a 
criterion  for  validating  selection,  classification  and  retention  policies  and  instruments. 

Tire  PC  construct  was  first  introduced  by  Carpenter,  Monaco,  O’Mara,  and  Teachout  (1989)  in  a  study 
attempting  to  relate  aptitude  and  experience  to  supervisor  estimates  of  productivity  in  one  Air  Force 
electronics  specialty  (328X0,  Avionics  Communications).  Their  objectives  were  to;  (1)  determine  the 
feasibility  and  validity  of  using  supervisor  estimates  of  relative  performance  time  as  a  measure  of 
performance;  (2)  estimate  the  extent  to  which  mental  aptitudes  affect  this  performance  measure;  and  (3) 
develop  and  apply  a  prototype  optimization  model  using  die  relationships  among  aptitude,  experience, 
productivity  and  cost  to  set  enlistment  standards.  Task  clusters  were  created  and  Subject  Matter  Expert 
(SME)  estimates  of  normal  (benchmark)  performance  times  for  each  cluster  woe  obtained.  Supervisors 
were  then  asked  to  rate  the  relative  performance  times  of  each  of  their  subordinates  on  each  of  the  task 
clusters.  The  relative  ratings  were  obtained  by  asking  the  supervisors  to  identify  one  of  their  subordinates 
as  performing  closest  to  the  benchmark  time  for  a  task  cluster,  and  then  estimating  how  long  it  would  take 
each  of  the  other  workers  to  accomplish  the  same  amount  of  work  as  the  benchmark  worker  could 
accomplish  in  an  hour. 
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For  construct  validation,  supervisors’  estimates  of  relative  performance  times  were  correlated  with  a 
varies  of  other  job  performance  measures.  Overall,  the  magnitude  of  the  obtained  correlations  were 
characterized  in  the  report  as  modest  The  procedure  used  to  solicit  supervisor  estimates  of  relative 
performance  times  may  have  contributed  to  the  low  validity.  Each  supervisor  in  the  sample  had  an 
average  of  only  4-5  subordinates  to  choose  from  in  selecting  a  benchmark  worker  against  which  to 
evaluate  their  other  workers.  If  the  benchmark  workers  selected  were  not  in  fact  "normal"  performers 
(quite  possible,  given  the  small  group  from  which  they  were  selected),  then  the  relative  time  estimates  for 
the  other  workers  would  be  biased.  A  more  systematic  method  for  establishing  stable  performance  time 
benchmarks  might  have  improved  the  validity  of  the  performance  time  estimates. 

The  second  objective  of  the  study  was  accomplished  with  more  success  than  the  first  A  regression 
model  was  developed  relating  job  experience  (in  months)  and  aptitude  (Electronic  composite  from 
ASVAB)  with  PC  An  R-squared  of  0.44  was  achieved,  which  the  authors  characterized  as  a  "reasonably 
good  fit"  (p.  23). 

To  address  the  third  objective  of  the  study,  die  regression  equation  was  used  in  an  optimization  model 
to  determine  the  electronic  cut  score  that  would  produce  die  minimum  cost  per  productive  unit  over  the 
first  term  of  service.  However,  because  no  cost  or  supply  constraints  for  high  quality  recruits  were 
included  in  the  model,  a  trivial  solution  was  obtained  (i.e.,  set  the  cut  score  at  the  maximum).  A  follow- 
on  study  (Faneuff,  Valentine,  Stone,  Curry,  &  Hageman,  1990)  expanded  the  optimization  process  by 
introducing  multiple  AFSs,  recruiting  market  constraints,  and  quality  cost  differentials  using  hypothetical 
data. 

A  study  conducted  by  Sacked,  Zedeck,  and  Fogli  (1988)  to  examine  the  relationship  between  maximum 
and  typical  performance  measures  is  also  of  interest  to  the  present  research.  They  postulated  that  job 
performance  criterion  measures  differ  along  a  continuum  of  typical  to  maximum  performance  with 
maximum  performance  measures  meeting  the  following  criteria: 

The  subjects  must  be  explicitly  aware  that  their  performance  is  being  evaluated 
Explicit  instructions  must  be  given  and  accepted  that  performance  is  to  be  maximized 
Performance  must  be  measured  over  a  short  enough  time  period  to  keep  the  subjects’  attention 
focused  on  the  goal  of  maximum  performance 
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They  also  hypothesized  that  typical  and  maximum  performance  levels  are  affected  by  different 
individual  characteristics;  therefore,  the  two  types  of  measures  should  not  be  highly  correlated.  Using  data 
collected  on  supermarket  check-out  clerks  measuring  task  performance  times  under  both  typical  and 
maximum  performance  scenarios,  the  authors  found  very  low  correlations  between  the  performance 
measures.  They  also  found  weak  relationships  between  measures  of  speed  and  accuracy.  This  work  is 
relevant  to  the  present  study  because  of  the  need  to  validate  the  PC  construct  (clearly  a  maximum 
performance  measure)  against  other  job  performance  measures  which  may  differ  along  the  typical- 
maximum  performance  measure  spectrum. 

Purpose  and  Approach 

The  present  study  was  intended  to  expand  on  the  first  objective  of  the  Carpenter  et  al.  study  and  to  try 
some  alternative  techniques  in  areas  that  were  not  fully  developed  in  the  earlier  study.  Before  PC  can  be 
integrated  into  the  Air  Force  personnel  management  system,  it  must  be  thoroughly  validated  as  a 
meaningful  measure  of  performance,  and  practical  procedures  for  routinely  collecting  the  necessary  data 
must  be  developed  and  tested.  To  this  end,  the  first  purpose  of  this  study  was  to  develop  procedures  and 
instruments  that  could  be  used  to  routinely  collect  PC  data  throughout  the  Air  Force.  In  particular,  an 
alternative  to  the  "benchmark  worker"  approach  to  anchoring  supervisor  estimates  was  sought  The 
procedures  and  instruments  were  then  evaluated  in  a  limited  field  test  of  selected  Air  Force  jobs. 

The  second  purpose  of  the  project  was  to  collect  both  PC  and  job  performance  data  on  a  sample  of 
airmen.  The  PC  data  were  collected  by  having  supervisors  estimate  how  long  each  of  their  subordinate 
airman  would  take  to  perform  each  of  a  representative  sample  of  tasks  from  their  AFS.  Obtaining  PC  data 
via  supervisor  estimates  offered  a  cost-effective  alternative  to  actually  timing  airmen  while  they  performed 
the  tasks.  However,  the  accuracy  of  supervisor  estimates  must  be  established  before  this  technique  can 
be  used,  operationally.  To  help  determine  the  relationship  between  estimated  and  actual  task  performance 
times,  some  of  the  airmen  in  the  sample  were  timed  while  performing  a  subset  of  the  sample  tasks. 

To  support  future  analyses  of  construct  validity,  an  alternative  job  performance  measure~a  task-specific 
written  job  knowledge  test  (JKT)-was  administered  to  the  airmen  to  determine  their  proficiency  in  a 
subset  of  the  tasks.  Comparing  PC  with  JKT  scores  may  be  complicated  by  the  fact  that  one  measure 
(PC)  focuses  on  speed  while  the  other  (JKT)  focuses  on  accuracy.  Similar  studies  (e.g.,  Sackett  et  al.. 
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1988)  have  typically  found  low  correlations  between  speed  and  accuracy  performance  measures.  The  PC 
and  JKT  measures  may  also  differ  on  Sackett’s  maximum  to  typical  performance  continuum.  PC  appears 
to  meet  Sackett’s  criteria  as  a  maximum  perfoimance  measure  while  the  JKT,  being  an  untimed  written 
test,  may  measure  more  typical  performance  levels.  Sackett  also  postulated  that  motivation  and  ability 
have  differential  effects  on  typical  and  maximum  performance.  Measures  of  job  interest  and  motivation 
were  also  collected  on  airmen  in  the  current  effort  to  control  for  these  effects  in  follow-on  construct 
validity  analyses. 

Using  these  data,  a  variety  of  analyses  can  be  conducted  to  determine  the  relationship  between 
estimated  and  actual  (timed)  PC  and  between  PC  and  job  knowledge,  controlling  for  motivation  and 
ability.  Potential  predictors  of  productive  capacity,  such  as  aptitude,  experience,  interest  and  motivation, 
can  also  be  analyzed.  Results  of  these  analyses  will  be  published  in  subsequent  reports. 

The  current  report  focuses  on  tire  data  collection  instruments  and  the  procedures  used  in  their 
administration.  The  final  two  sections  comment  on  the  methodological  issues  encountered  in  the  project 
and  offer  suggestions  for  future  research. 

H.  METHODOLOGY 
Air  Force  Specialty  (AFS)  Selection 

Early  in  the  project  the  decision  was  made  to  limit  the  initial  field  test  of  the  productive  capacity 
measurement  methodology  to  four  AFSs.  Enlisted  jobs  in  the  Air  Force  are  categorized  into  one  of  four 
aptitude  areas  (Mechanical,  Administrative,  General,  or  Electronic).  One  AFS  from  each  aptitude  area  was 
desired  to  provide  coverage  of  the  range  of  Air  Force  jobs.  Further,  four  AFSs  were  judged  to  be 
sufficient  for  evaluating  the  suitability  of  the  methodology  for  application  on  a  large  scale  to  numerous 
Air  Force  jobs  in  follow-on  efforts. 

The  specialties  were  chosen  from  among  the  eight  previously  studied  under  the  Air  Force’s  JPMS 
project  to  take  advantage  of  the  in-depth  task  analysis  information  accumulated  (Laue,  Hedge,  Wall, 
Pederson,  &  Bentley,  1989).  The  AFSs  studied  late  in  the  JPMS  project  were  preferred  to  assure  the 
currency  of  task-level  data.  Moreover,  for  those  specialties  written  JKTs  existed  (Laue  et  al.,  1989)  that 
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measured  knowledge  on  a  representative  sample  of  tasks.  The  availability  of  JKTs  was  an  important 
consideration  for  the  current  effort  An  alternate  job  performance  measure  was  needed  to  evaluate  the 
construct  validity  of  the  productive  capacity  measure.  In  a  previous  study  of  the  JPMS  measures,  the 
JKTs  had  been  shown  to  load  on  the  same  factor,  labeled  Technical  Proficiency,  as  hands-on  performance 
measures  (Dickinson  &  Teachout,  1991).  A  final  issue  in  selecting  AFSs  for  die  current  project  concerned 
subject  availability.  The  JPMS  experience  was  helpful  in  identifying  AFSs  with  sufficient  numbers  of  job 
incumbents  to  ensure  that  reasonable  sample  sizes  could  be  achieved. 

After  taking  into  account  these  considerations,  the  following  AFSs  were  selected: 

122X0,  Aircrew  Life  Support  (General  aptitude  requirement) 

454X1,  Aerospace  Ground  Equipment  (Mechanical  aptitude  requirement) 

455X2,  Avionic  Communication  and  Navigation  Systems  (Electronic  aptitude  requirement) 
732X0,  Personnel  (Administrative  aptitude  requirement) 

Note  that  455X2  appeared  under  the  title  328X0  (Avionic  Communications  Specialist)  in  the  JPMS 
project.  The  new  455X2  title  reflects  die  Rivet  Workforce  combination  of  328X0,  328X1,  and  328X44. 
AFS  455X2  was  chosen  over  a  more  recendy  studied  electronic  aptitude  AFS  (324X0,  Precision 
Measurement  Equipment  Laboratory),  even  though  the  more  current  AFS  had  a  JKT.  The  choice  was 
made  because  455X2  has  significantly  more  people  assigned  than  does  324X0,  thus  increasing  the  chances 
of  achieving  the  desired  sample  sizes. 


Task  Selection 

Once  the  AFSs  were  identified,  specific  tasks  in  each  AFS  had  to  be  selected  for  performance  time 
estimation  and  measurement  It  was  important  that  the  tasks  be  routinely  performed  by  most  first-term 
airmen  in  the  AFS,  regardless  of  command,  base  or  duty  section  of  assignment  The  Walk  Through 
Performance  Test  (WTPT)  Phase  I5  tasks  used  in  the  JPMS  project  met  this  criterion  (Hedge  &  Lipscomb, 


4  Rivet  Workforce  is  an  Air  Force  initiative  to  combine  aircraft  maintenance  AFSs,  thus  reducing 
the  number  of  technicians  required  to  support  aircraft  deployments. 

5  Phase  n  tasks  are  specific  to  only  one  command  or  duty  section  within  the  AFS. 
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1987),  and  had  the  added  benefit  of  being  fully  articulated-- that  is  detailed  assumptions,  background 
information,  tools  and  equipment  needs,  and  instructions  to  the  test  administrator  were  well  documented 
in  the  WTPT. 

Unfortunately,  there  were  not  enough  Phase  I  WTPT  tasks  available  to  adequately  evaluate  productive 
capacity  measurement  across  a  wide  variety  of  duties  (11  for  Aircrew  Life  Support  (122X0),  8  for 
Personnel  (732X0),  26  for  Aerospace  Ground  Equipment  (454X1),  and  16  for  Avionic  Communication 
and  Navigation  Systems  (455X2)).  Thus,  it  was  necessary  to  supplement  the  WTPT  tasks  from  the 
complete  task  inventory  for  each  AFS.  The  inventories  were  obtained  from  the  Occupational  Measurement 
Squadron  (OMS)  at  Randolph  AFB,  Texas.  Non- WTPT  tasks  were  chosen  from  the  inventories  based  on 
the  percent  of  first-term  airmen  performing  them,  again  to  increase  the  probability  that  airmen  in  the  field 
would  be  familiar  with  the  tasks.  Tasks  for  which  it  would  obviously  be  difficult  to  estimate  performance 
times  were  omitted,  especially  open-ended  tasks  like  "Conduct  On-the-Job  Training"  or  "Write 
Correspondence."  This  process  resulted  in  an  initial  sampling  of  tasks  as  follows  (totals  include  the  WTPT 
tasks): 


122X0  -  50  Tasks 
454X1  -  55  Tasks 
455X2  -  41  Tasks 
732X0  -  38  Tasks 

The  tasks  in  three  of  the  four  AFSs  tended  to  represent  certain  duty  areas  more  than  others,  probably 
because  these  duty  areas  contained  the  functions  most  frequently  performed  by  first-term  airmen.  For  AFS 
454X1  the  tasks  represented  flight  line  Aerospace  Ground  Equipment  (AGE)  as  opposed  to  mobile  AGE; 
in  AFS  455X2  the  tasks  came  primarily  from  the  repair  shop  rather  than  the  flight  line;  and  in  AFS  732X0 
the  Consolidated  Base  Personnel  Office  (CBPO)  functions  were  more  heavily  represented  than  were 
orderly  room  functions.  The  122X0  tasks  seemed  to  cover  die  spectrum  of  functions  in  the  career  field. 

Benchmark  Performance  Times 

The  next  step  in  the  methodology  was  to  establish  a  set  of  benchmark  performance  times  for  each  task 
in  each  specialty.  The  benchmark  times  were  necessary  to  meet  two  objectives:  (1)  to  give  supervisors 
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in  the  field  standard  reference  times  against  which  to  estimate  performance  times  for  their  subordinates 
(i.e.,  to  keep  the  time  estimation  process  from  being  totally  open-ended);  and  (2)  to  establish  the  fastest 
possible  performance  times  for  each  task  to  serve  as  the  T*  value  in  computing  PC  indices.  To  obtain 
the  time  benchmarks,  experienced  Non-Commissioned  Officers  (NCOs)  from  each  AFS  reviewed  the  tasks 
and  provided  their  best  judgments  as  to  the  range  of  times  the  task  might  take. 

Four  AFS  workshops  to  establish  the 
benchmark  times  were  scheduled  during 
December  1990  and  January  1991  at  Brooks 
AFB,  Texas.  Six  Subject  Matter  Experts 
(SMEs)  from  each  AFS  from  bases  in  the  San 
Antonio  area  were  invited  to  attend  each  2- 
day  workshop.  Characteristics  of  the  SMEs 
are  shown  in  Table  1.  Certain  ground  rules 
were  established  early  in  each  workshop  to 
ensure  consistency.  First,  definitions  of 
fastest,  normal  and  slowest  performance  times 
were  presented  to  the  SMEs.  The  definitions 
reflected  the  maximum  output  nature  of 
productive  capacity  by  emphasizing  speed, 
while  maintaining  an  acceptable  quality  level; 
they  were  as  follows: 

Fastest  Performance  Time:  The  time  it 
would  take  the  most  proficient  first-term  airman  in  the  AFS  to  perform  die  task  to  an  acceptable 
level  of  quality,  while  working  as  quickly  as  possible. 

Normal  Performance  Time:  The  time  it  would  take  the  average,  or  typical,  first-term  airman  in  the 
AFS  to  perform  the  task  to  an  acceptable  level  of  quality  while  working  as  quickly  as  possible. 

Slowest  Performance  Time:  The  maximum  time  a  supervisor  would  allow  a  first-term  airman  in 
the  AFS  to  work  on  the  task  before  stopping  him  or  her  and  assigning  die  task  to  someone  else. 


Table  1.  Subject  Matter  Expert  (SME) 
Characteristics 
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A  variant  of  the  Nominal  Group  Technique  (NGT)  for  obtaining  group  consensus  (Fox,  1989)  was  used 
during  the  workshops  to  elicit  SME  benchmark  time  estimates.  The  first  step  in  die  process  involved 
presenting  the  task  descriptions  to  die  SMEs  in  written  form  and  having  them  independently  make  their 
time  estimates,  with  no  discussion,  interaction  with  the  group,  or  further  instruction  from  the  workshop 
administrators.  Next,  the  six  independent  estimates  were  placed  on  transparencies  (one  per  task)  and 
displayed  for  review  by  die  group.  Each  SME,  in  turn,  had  an  opportunity  to  provide  verbal  rationale 
(without  debate)  for  his  or  her  estimate.  SMEs  could  then  revise  their  estimates  after  hearing  the  others’ 
rationale,  if  they  so  desired.  If  this  process  did  not  converge  on  a  consensus  estimate,  a  secret-ballot  vote 
was  taken  on  the  two  or  three  most  common  estimates  to  determine  consensus.  This  process  was  repeated 
for  each  benchmark  estimate  (fastest,  normal  and  slowest)  for  each  task. 

During  the  workshops  the  SMEs  assigned  benchmark  times  to  WITT  and  non- WITT  tasks  separately. 
Within  these  categories,  the  presentation  order  was  random.  First  the  tasks  were  grouped  by  duty  category 
(using  a  code  provided  in  the  task  inventory).  Then  die  order  of  duty  categories  was  randomly  selected, 
and  then  the  tasks  within  each  duty  category  were  randomly  sorted.  Tasks  were  presented  to  the  SMEs 
by  duty  category  so  they  could  focus  on  one  type  of  work  at  a  time  as  they  made  their  estimates. 

Some  features  of  the  methodology  were  refined  over  the  course  of  the  workshops.  At  the  first 
workshop.  Avionics  Communication  and  Navigation  System  (4SSX2),  SMEs  independently  assigned 
slowest,  normal,  and  fastest  times  to  the  non- WITT  tasks  first  Next  they  independently  assigned 
benchmark  times  to  the  WITT  tasks.  Then  die  NGT  procedure  was  used  to  achieve  group  consensus  for 
tire  three  benchmark  times  for  each  task.  However,  when  trying  to  achieve  consensus  for  the  non- WTPT 
tasks,  SMEs  had  to  add  constraints  and  assumptions  to  many  of  the  tasks  before  they  could  agree  on 
performance  times— some  of  the  tasks  were  simply  too  broadly  worded  to  permit  reasonable  performance 
time  estimates. 

For  this  first  workshop,  the  independently  estimated  performance  times  on  non-WITT  tasks,  assigned 
before  group  discussions,  do  not  have  much  meaning  since  SMEs  might  have  defined  the  tasks  in  different 
ways.  At  the  second  workshop  (Aerospace  Ground  Equipment  (454X1)),  the  non-WITT  tasks  were 
discussed  before  any  attempt  was  made  to  assign  performance  time  estimates.  During  these  discussions 
the  SMEs  agreed  on  various  constraints  and  assumptions  regarding  each  task,  such  as  how  many 
repetitions  of  a  procedure  would  be  done,  on  what  specific  equipment  the  task  would  be  performed,  what 
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tools  and  materials  would  be  available  to  the  performer,  and  at  which  points  in  a  procedure  the  task  being 
measured  would  start  and  stop. 

These  discussions  before  assigning  independent  performance  times  were  somewhat  helpful,  but 
subsequent  discussion  during  the  NGT  sessions  indicated  that  the  constraints  decided  upon  before 
assigning  performance  times  were  not  always  remembered  or  consistently  used  as  guides  when  assigning 
times  to  die  tasks. 

At  the  third  and  fourth  workshops,  Aircrew  Life  Support  (122X0)  and  Personnel  (732X0),  the  non- 
WTPT  tasks  were  again  discussed  first,  but  this  time  hard  copies  of  the  outcome  of  the  discussions  (i.e., 
task  descriptions  with  the  agreed-upon  constraints  and  assumptions  noted)  were  made  for  each  SME  to 
refer  to  when  assigning  their  independent  performance  time  estimates. 

Based  on  recommendations  from  the  SMEs  at  the  workshops,  some  tasks  were  eliminated  from  further 
data  collection  activities  because  they  were  either  no  longer  performed  in  the  field,  were  redundant  with 
another  task,  or  were  considered  too  trivial  for  timing.  After  the  workshops  the  task  totals  were  revised 
as  follows:  122X0,  from  50  to  45;  454X1,  from  55  to  50, 455X2,  no  change  at  41;  and  732X0,  from  38 
to  36.  These  were  the  numbers  of  tasks  taken  to  the  field  for  time  estimation  by  supervisors. 

Field  Data  Collection 

Purpose 

If  the  Air  Force  is  to  ultimately  have  a  productivity-based  personnel  planning  and  management  system, 
practical  procedures  and  instruments  must  be  developed  for  routinely  collecting  PC  information  on  large 
numbers  of  airmen  in  a  broad  cross-section  of  AFSs.  Therefore,  the  primary  purpose  of  the  field  data 
collection  phase  of  this  project  was  to  evaluate  the  instruments  and  procedures  described  below  under 
realistic,  operational  conditions.  Rather  than  collecting  data  in  a  controlled  laboratory  environment,  visits 
were  mad*  to  active  Air  Force  bases  representing  the  full  range  of  operational  flying  missions,  including 
Strategic  Air  Command  (SAC),  Tactical  Air  Command  (TAQ,  and  Military  Airlift  Command  (MAC),  plus 
a  support  mission.  Air  Force  Systems  Command  (AFSC).  Data  collection  procedures  were  applied  as  they 
would  be  under  "live"  conditions,  using  real  tasks  and  airmen  in  their  actual  work  environment  While 
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this  approach  reduced  control  over  potential  errors  and  sources  of  variance  in  the  data  that  were  collected, 
it  provided  a  more  realistic  evaluation  of  the  proposed  procedures  and  instruments. 

The  secondary  purpose  of  the  field  visits  was  to  collect,  as  precisely  as  possible  under  field  conditions, 
estimated  and  actual  task  performance  times,  task  knowledge  information,  and  motivation  and  interest 
measures  for  use  in  future  analytical  studies  to  evaluate: 

The  accuracy  with  which  supervisors  can  estimate  task  performance  times  for  their  subordinates 

The  validity  of  PC  as  a  measure  of  job  performance 

The  effects  of  aptitude,  experience,  motivation  and  interest  on  PC 

Air  Force  Base  Selection 

Collection  of  PC  data  requires  direct  access  to  airmen,  whose  productive  capacity  is  to  be  measured, 
and  to  their  supervisors  (who  provide  task  performance  time  estimates  on  their  subordinates),  as  well  as 
access  to  their  work  centers  to  conduct  actual  hands-on  task  timing  measurement  The  set  of  bases 
selected  required  an  adequate  number  of  potential  subjects  to  achieve  desired  sample  sizes,  while  leaving 
enough  airmen  and  supervisors  on  duty  to  continue  day-to-day  operations  during  the  testing  periods. 

The  base  selection  procedure  involved  the  use  of  frequency  distributions  of  personnel  in  the  AFSs  of 
interest  at  every  Air  Force  base.  These  distributions  were  created  using  the  December  1990  extract  of  the 
Uniform  Airman  Record  (UAR),  the  master  computer  file  maintained  by  the  Air  Force  Military  Personnel 
Center  (AFMPC)  at  Randolph  AFB,  Texas  on  every  active  duty  airman.  The  UAR  contains  numerous 
variables  on  each  individual,  including  base  of  assignment  and  AFS,  data  fields  that  were  used  in  creating 
the  frequency  distributions. 

Bases  with  predominantly  technical  training  missions  (like  Keesler  AFB  in  Mississippi)  were  excluded 
from  consideration.  Many  of  the  potential  subjects  at  these  bases  were  trainees  who  were  not  yet  qualified 
in  their  AFSs.  Other  bases  were  rejected  because  their  missions  were  classified,  thus  restricting  access 
to  airmen,  their  supervisors  and  equipment  in  their  work  centers.  Bases  outside  the  continental  U.S.  were 
also  excluded  to  stay  within  the  project’s  travel  budget 


After  these  constraints  were  imposed,  distributions  for  the  remaining  bases  indicated  that  10  bases  was 
the  minimum  number  needed  to  satisfy  study  requirements.  The  bases  selected  were:  Travis  AFB, 
California  (MAC);  Beale  AFB,  California  (SAC);  George  AFB,  California  (TAC);  Davis-Monthan  AFB, 
Arizona  (TAC);  Holloman  AFB,  New  Mexico  (TAC);  Langley  AFB,  Virginia  (TAC);  Shaw  AFB,  South 
Carolina  (TAC);  Offutt  AFB,  Nebraska  (SAC);  Eglin  AFB,  Florida  (TAC  and  AFSC);  and  McGuire  AFB, 
New  Jersey  (MAC).  McGuire  AFB  was  subsequently  dropped  from  the  study  by  MAC  because  of 
DESERT  STORM  deployments  and  added  workload,  and  Charleston  AFB,  South  Carolina  was  substituted. 

Subject  Selection 

A  sample  size  of  200  subjects  per  AFS  was  targeted  because  this  was  die  maximum  number  that  could 
reasonably  be  tested  with  the  time  and  research  personnel  available  for  this  project  Also,  a  sample  of  this 
magnitude  would  permit  reasonable  confidence  in  the  planned  statistical  analyses,  even  with  some  shortfall 
in  airman  availability  during  the  field  test 

Subjects  for  each  AFS  were  selected  to  be  representative  of  base  populations  in  terms  of  job 
experience,  race,  and  gender.  Job  experience  was  an  important  selection  factor,  because  it  was  postulated 
to  contribute  to  productive  capacity;  more  experienced  airmen  were  expected  to  have  higher  work  output 
An  effort  was  made  to  sample  subjects  over  a  range  of  job  experience  while  still  focusing  on  those  who 
typically  perform  the  hands-on  technical  tasks  being  measured  (as  opposed  to  supervisory  or  managerial 
tasks).  The  Air  Force  codes  the  amount  of  training,  expertise  and  experience  an  individual  has  in  a 
particular  AFS  by  awarding  a  skill  level  ranging  from  0  to  9,  with  skill  level  0  reserved  for  Chief  Master 
Sergeants  and  skill  level  1  indicating  an  entry-level  trainee  -operationally  only  values  of  3,  5,  7,  and  9 
are  used6.  Because  current  data  on  the  actual  number  of  months  and  years  an  individual  has  on  a 
particular  job  were  difficult  to  obtain  from  the  computerized  personnel  records,  skill  level  was  used  as  a 
surrogate  for  experience.  Thus,  the  sample  was  limited  to  airmen  with  a  3  or  5  skill  level  and  less  than 
six  years  in  the  Air  Force.  Prior  Air  Force  studies  of  the  relationship  between  job  performance  and 
experience  have  limited  their  observations  to  first-term  airmen,  those  with  four  or  fewer  years  of  service 
(Alley  &  Teachout,  1990;  Carpenter  et  al.,  1989;  Lance.  Hedge,  &  Alley,  1987).  An  interest  in  the  current 


6  Skill  level  is  recorded  as  the  forth  digit  of  an  individual’s  Air  Force  Specialty  Code  (e.g., 
45532).  The  required  skill  level  for  each  enlisted  position  is  similarly  coded  in  the  manpower  files. 
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project  was  the  utility  of  productive  capacity  for  force  structure  planning.  Decisions  about  experience 
mixes  needed  to  maintain  force  productivity  are  most  germane  to  career  airmen,  those  with  more  than  four 
years  of  service.  The  six  years  of  service  limitation  imposed  on  samples  for  this  project  provided  the 
opportunity  for  modeling  performance-experience  relationships  beyond  the  first  term,  while  ensuring  that 
airmen  identified  as  study  participants  still  performed  technical-level  tasks.  Samples  were  also  stratified 
by  race  and  gender. 

Test  subjects  were  actually  selected  by  the  Table  2.  Sample  Distribution 

participating  bases,  using  criteria  provided  by  the 
research  team,  because  the  bases  had  more  current 
information  on  manning,  deployments  and 
personnel  status.  Each  base  was  provided  a 
detailed  matrix  indicating  the  number  of  subjects 
required  by  AFS,  skill  level,  gender  and  race 
based  on  the  distributions  of  these  factors  at  the 
base  as  of  December  1990.  The  bases  were  also 
verbally  requested  to  select  airmen  from  particular 
duty  areas  in  three  of  the  four  AFSs  (454X1, 

455X2  and  732X0)  because  the  tasks  selected  for 
measurement  in  these  AFSs  tended  to  cluster  in 
certain  duty  areas.  Because  the  bases  did  the 
actual  subject  selection,  it  was  not  possible  to 
stratify  the  sample  by  aptitude  level  (even  though 
this  would  have  been  desirable)  since  ASVAB 
scores  are  not  available  in  base-level  personnel  records. 

After  the  subjects  were  selected  by  the  bases,  sub-samples  of  60  subjects  per  AFS  (6  per  base)  were 
identified  to  participate  in  the  actual  task  timing  studies.  Again,  these  subjects  were  selected  to  represent, 
to  the  extent  possible,  the  base/AFS  population  with  respect  to  skill  level,  race  and  gender.  Table  2 
summarizes  the  target  and  actual  sample  distributions.  Overall  sample  size  objectives  were  achieved  in 
two  AFSs  (454X1  and  732X0)  but  fell  about  25  percent  short  in  the  other  two  AFSs  due  to  unavailability 
of  personnel  at  the  test  sites.  Junior  personnel  (3-levels)  are  substantially  over  represented  in  the  122X0 


122X0 

454X1 

455X2 

732X0 

Goal / 
Actual 

Goal/ 

Actual 

Goal / 
Actual 

Goal/ 

Actual 

N 

200/153 

200/195 

200/144 

200/186 

SKILL 

LVL 

% 

% 

% 

% 

3 

29/54 

48/45 

34/44 

28/31 

5 

71/44 

52/55 

66/56 

72/68 

7 

0/2 

(to 

0/<l 

on 

RACE 

% 

% 

% 

% 

Cbuc 

70/65 

81/81 

89/89 

55/54 

Mia 

30/35 

19/19 

11/11 

45/46 

SEX 

% 

* 

% 

% 

Male 

91/86 

94/94 

92/91 

50/44 

Fem 

9/14 

6/6 

8/9 

50/56 
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of  personnel  at  the  test  sites.  Junior  personnel  (3-levels)  are  substantially  over  represented  in  the  122X0 
and  4SSX2  samples,  slightly  over  represented  in  732X0  and  somewhat  under  represented  in  454X1.  A 
total  of  six  7  skill-level  personnel  were  also  tested.  The  racial  composition  is  proportional  to  the 
population  distribution  except  in  the  122X0  sample,  where  minorities  are  slightly  over  represented. 
Sample  gender  distributions  are  similar  to  the  population  distribution  in  the  454X1  and  455X2  samples; 
however,  females  are  over  represented  in  the  122X0  and  732X0  samples. 

Pilot  Test 


Prior  to  the  first  field  data  collection  visit,  a  pilot  test  of  all  draft  data  collection  instruments  and 
procedures  was  conducted  at  Andrews  AFB,  Maryland.  The  primary  purpose  of  the  test  was  to  try  out 
each  form,  instrument,  instruction  set  and  procedure  on  a  small  sample  of  airmen  and  their  supervisors 
to  identify  problems,  resolve  inconsistencies  and  to  develop  a  workable  sequence  and  schedule  of  events 
for  the  actual  field  visits.  A  second  purpose  was  to  familiarize  the  data  collection  team  with  the 
environment  in  which  they  would  be  working  and  to  allow  them  to  practice  the  procedures  under  realistic 
conditions.  The  pilot  test  proved  invaluable  by  surfacing  problems  and  issues  prior  to  the  first  actual  data 
collection  visit  Numerous  refinements  to  instructions,  data  collection  forms,  and  scales  were  made  as  a 
result 

Instruments 

A  combination  of  existing  and  newly-developed  forms  and  instruments  were  used  to  collect  data  at  the 
ten  bases.  Three  categories  of  data  collection  materials  were  required  to:  (1)  Collect  background 
information  on  supervisors,  and  record  their  estimates  of  task  performance  times  for  their  subordinates; 
(2)  obtain  background  information,  job  knowledge,  interest  and  motivation  measures  on  the  airmen  for 
whom  supervisor  time  estimates  were  obtained;  and  (3)  collect  actual  task  performance  times  on  sub¬ 
samples  of  airmen  and  tasks.  Table  3  lists  the  forms  and  instruments,  indicating  which  were  developed 
specifically  for  this  project  and  which  were  taken  from  prior  work.  Each  will  be  described  in  more  detail 
in  the  following  sections. 


Supervisor  Ratine  Forms 

The  supervisors  in  each  AFS  were  provided  with  four  data  collection  instruments,  each  of  which  was 
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Table  3.  Data  Collection  Forms  and  Instruments 


developed  specifically  for  this  project  The 
Supervisor/ Airman  List  given  to  each  supervisor 
identified  the  individuals)  for  whom  the 
supervisor  was  to  provide  time  estimates.  This 
form  also  served  as  a  data  collection  instrument  in 
that  it  asked  for  the  number  of  months  the 
supervisor  had  supervised  each  airman.  The  form 
also  served  as  a  cross-check  on  the  airman  testing 
sessions,  identifying  individuals  who  were  not 
present  or  who  were  substituting  for  another 
airman.  A  copy  of  this  form  is  in  Appendix  A. 


The  Background  Information  Form  contained 
the  required  Privacy  Act  statement,  citing  the 
authority  for  collecting  the  data,  the  project’s 
purpose  and  use  of  the  information,  and  the 
voluntary  nature  of  participation.  The  form  was  also  used  to  collect  identification  data  on  the  supervisors, 
including  name.  Social  Security  number,  grade,  AFS,  base  and  command  of  assignment,  duty  title,  and 
active/reserve  status.  The  most  important  use  of  this  form  was  to  collect  a  series  of  experience  variables, 
including  total  time  in  the  Air  Force,  time  in  AFS,  time  in  grade,  time  in  current  position,  and  time  as  a 
supervisor.  With  the  exception  of  total  time  in  the  Air  Force  and  time  in  grade,  the  experience 
information  was  not  available  in  historical  computerized  personnel  files,  and  thus  had  to  be  collected 
directly  from  the  supervisors.  A  copy  of  the  Background  Information  Form  is  in  Appendix  B. 

The  Tasks  for  Time  Estimation  Booklets  began  with  a  set  of  instructions  inside  the  front  cover,  which 
the  test  administrator  read  and  discussed  with  die  supervisors  (see  Appendix  C-I).  The  descriptions  of 
the  tasks  began  on  die  next  page  of  the  booklet  Because  of  their  bulk,  the  actual  booklets  for  each  AFS 
are  not  included  in  this  report  although  they  are  available  from  the  project  sponsor.  Examples  of  WTPT 
and  non- WTPT  tasks  are  shown  in  Appendix  C-2  and  C-3,  respectively,  for  the  Aircrew  Life  Support 
specialty  to  illustrate  the  differences  in  level  of  detail. 


New/Exfattac 

Supervisor  Rating  Forms: 

•  Supervisor/ Airman  List 

New 

•  Badcgrmmd  Information 

New 

Form 

•  Tasks  for  Time 

New 

.  Time  Estimation  Form 

New 

Airman  Testing  Forms: 

•  Backgratmd  Information 

Same  as  Supervisors' 

Form 

•  Vocational  Interest  for 

Existing 

Career  Enhancement 

•  Generalized  Motivation 

Existing 

Scale 

•  Job  Knowledge  Test 

Existing 

•  Scaanatde  Answer  Sheet 

New 

Task  Timing  Forms: 

•  Tasks  for  Timing 

New 

Booklets 

•  Task  Timing  Forms 

New 
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In  the  booklets,  tasks  taken  from  the  WTPT  were  listed  first,  followed  by  the  non-WTPT  tasks;  within 
each  category  (WTPT  and  non-WTPT),  the  tasks  were  ordered  by  task  number.  The  task  numbers  were 
taken  from  die  Air  Force  master  task  inventory— they  each  began  with  an  alpha  prefix  indicating  the 
general  work  category  (e.g.,  in  Aircrew  Life  Support,  die  "H"  prefix  indicates  fitting  and  maintaining 
helmets),  followed  by  a  sequence  number  of  up  to  four  digits.  In  a  few  cases  an  alpha  suffix  ("A"  or  "B") 
was  added  to  tasks  that  had  been  divided  into  two  tasks  by  the  SMEs  at  the  benchmark  workshops. 

The  WTPT  tasks  were  headed  by  the  task  number  and  title.  Following  were  the  benchmark  time 
estimates  developed  at  the  SME  workshops.  The  tools  and  equipment  needed  to  perform  the  task, 
background  on  the  task,  the  setup  configuration,  instructions  for  the  task  timing  administrator,  and 
instructions  for  die  airmen  being  timed,  were  also  listed  to  help  supervisors  envision  the  environment  in 
which  the  airman  would  be  working.  This  information,  along  with  the  step-by-step  description  of  the  tasks 
which  followed,  attempted  to  give  the  supervisors  making  performance  time  estimates  as  complete  a 
picture  as  possible  of  the  work  environment  and  any  constraints  the  airmen  would  have  to  overcome. 

Following  the  WTPT  tasks  in  the  booklets  were  the  non-WTPT  tasks.  These  were  also  headed  by  the 
task  number  and  title  and  included  the  benchmark  time  estimates  and  any  additional  assumptions  or  steps 
that  were  generated  by  the  SMEs  at  the  workshops.  Considerably  less  detail  was  available  on  these  tasks 
compared  to  the  WTPT  tasks. 

The  final  form  used  by  the  supervisors  was  die  Time  Estimation  Form,  on  which  they  recorded  their 
estimates.  This  form  identified  die  particular  AFS,  followed  by  spaces  for  the  supervisor’s  name  and  the 
name  and  Social  Security  Number  of  the  airman  being  rated.  The  far  left-hand  column  of  the  form 
contained  the  task  numbers,  in  the  same  order  as  presented  in  the  Tasks  for  Time  Estimation  booklets. 
The  second  column  contained  a  set  of  three  boxes.  Here  the  supervisor  placed  a  check  for  how  often  they 
had  observed  the  airman  perform  the  task  in  question.  The  boxes  were  labeled  as  "RH  for  regularly,  "O” 
for  occasionally,  and  "N"  for  never,  following  the  same  scale  used  by  Carpenter  et  al.  (1989).  The  next 
column  was  a  time  line  which  graphically  presented  the  benchmark  time  estimates,  presented  in  die  order 
of  fastest,  normal,  and  slowest  The  final  far  right-hand  column  of  the  form  contained  another  set  of  three 
boxes  labelled  "Hr"  for  hours,  "Min"  for  minutes,  and  "Sec"  for  seconds.  It  is  here  that  the  supervisors 
were  asked  to  record  how  long  the  airman  would  take  to  perform  the  task  at  a  satisfactory  level,  while 
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working  as  quickly  as  possible.  Figure  1  shows  the  top  of  the  first  page  of  the  Aircrew  Life  Support  Time 
Estimation  Form  to  illustrate  the  frequency  and  time  scales. 


AFS  122X0 


Name 


Airman's  Name 


Airman's  SSN 


R  -  Regularly 
O  «  Occasionally 
N  -  Never 

How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest" 


Hr  ■  Hours 
Min  •  Minutes 
Sec  »  Seconds 

What  is  Incumbent's 
Performance 
Time? 

(Fill  in  box  below) 


El  99 


R 

0 

N 

2  min 
30  sec 


4  min 


7  min 


Hr 

Min 

Sec 

Figure  1.  Time  Estimation  Form  Example 


At  the  end  of  each  Time  Estimation  Form  was  a  scale  on  which  die  supervisors  were  asked  to  provide 
an  overall  productivity  rating  for  the  airman  being  rated.  The  instructions  and  measurement  scale  are 
shown  in  Figure  2.  Copies  of  the  Time  Estimation  Forms  for  each  of  the  four  AFSs  can  be  found  in 
Appendix  D. 

In  this  specialty,  consider  the  maximum  amount  of  acceptable  work  that  can  be 
done  by  a  person  on  a  typical  day  as  100  percent.  What  percent  of  maximum 
could  the  person  you  are  currently  rating  do  in  a  typical  day?  Write  your 
estimate  in  the  box  below. 


Percent  of 
Maximum 


Figure  2.  Overall  Productivity  Rating  Scale 
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Airman  Testing  Foims 


While  the  supervisors  were  estimating  task  performance  times  for  their  subordinate  airmen,  the  airmen 
were  being  tested  in  a  group  in  another  room.  The  first  form  they  were  asked  to  fill  out  was  the 
Background  Information  Form-the  same  form  used  in  the  supervisor  rating  sessions.  This  form  was 
especially  critical  for  the  airmen  because  it  was  the  only  source  of  experience  data  (time  in  AFS),  which 
was  postulated  to  be  a  predictor  of  productive  capacity. 

Job  Knowledee  Test 

One  of  die  key  issues  regarding  supervisor  estimates  of  productive  capacity  is  how  they  relate  to  other 
measures  of  job  performance.  Thus,  one  of  the  objectives  of  this  study  was  to  collect  job  performance 
data,  on  the  same  airmen  for  whom  productive  capacity  ratings  would  be  computed  as  a  means  of 
validating  the  PC  construct  While  actual  hands-on  work  sample  tests  are  generally  viewed  as  die  highest 
fidelity  measures  of  job  performance,  their  administration  is  extremely  time  consuming  and  expensive 
(Hedge  &  Teachout,  1986).  Since  the  data  collection  phase  of  this  study  was  already  quite  protracted  and 
complex,  a  simpler  surrogate  for  the  hands-on  performance  measure  was  sought  The  DoD  has  developed 
several  surrogate  performance  measures  as  more  feasible  alternatives  to  hands-on  performance  testing,  one 
of  which  is  the  task-specific,  pencil-and-paper  job  knowledge  test  (Bentley,  Ringenbach,  &  Augustin, 
1989).  Since  JKTs  had  previously  been  developed  for  3  of  the  4  AFSs  in  this  study,  and  because  they 
are  relatively  quick  and  easy  to  administer  and  score,  they  were  selected  as  the  surrogate  performance 
measure  to  be  used. 

The  three  JKTs  were  developed  by  the  Air  Force  to  demonstrate  the  transferability  of  an  Army  job 
performance  measurement  technology  to  another  service  (Bentley  et  al.,  1989).  Using  the  Army 
methodology,  the  Air  Force  based  the  JKTs  on  the  specific  tasks  used  in  the  WTPTs  developed  as  part 
of  the  JPMS  project  The  WTPT  tasks  were  presented  to  groups  of  SMEs  from  each  AFS,  who  were 
asked  to  identify  the  steps  that  comprise  die  key  elements  of  successful  performance  of  each  task.  Test 
items  were  then  written  for  each  key  element  concentrating  on  the  application  of  die  knowledge  needed 
to  successfully  perform  the  task.  All  items  were  written  in  a  multiple-choice  format  with  two  to  five 
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alternative  responses  per  item-the  number  of  alternatives  being  directly  related  to  the  number  of  plausible 
incorrect  responses.  Alternatives  were  written  such  that  only  one  was  correct  The  number  of  items  per 
task  ranged  from  two  to  eighteen. 

A  second  group  of  SMEs  was  then  convened  for  each  AFS  to  review  the  items  and  responses  for 
technical  accuracy  and  correct  wording.  This  was  followed  by  a  pilot  test  of  each  draft  JKT  using  a  small 
sample  of  first-term  airmen  in  foe  field.  After  incorporating  improvements  and  corrections  from  the  pilot 
tests,  the  JKTs  woe  administered  to  larger  groups  of  airmen  (25-50)  to  obtain  item  response  data. 
Extremely  easy  and  difficult  items,  those  with  passing  rates  below  10  percent  or  above  90  percent,  were 
eliminated,  as  were  items  with  low  item-total  score  correlations,  showing  poor  discrimination  of  airmen 
job  knowledge  levels.  Additional  marginal  items  were  eliminated  to  keep  foe  test  administration  time 
under  two  hours  for  each  JKT. 

After  all  revisions  were  made,  foe  final  version  of  foe  122X0  JKT  prepared  for  foe  JPMS  project 
(Phase  I,  or  common  tasks  only)  covered  8  tasks  with  70  items  (4-19  items  per  task);  alpha  coefficients 
for  foe  task  tests  ranged  from  -.03  to  .68  (mean  of  .34).  The  final  version  of  foe  454X1  JKT  used  in  this 
study  (booklet  B)  covered  14  tasks  with  75  items  (2-11  items  per  task);  its  alpha  coefficients  ranged  from 
-51  to  .42  (mean  of  .17).  The  732X0  JKT  (again.  Phase  I  common  tasks  only)  included  8  tasks  with  47 
total  items  (2-10  items  per  task);  alpha  coefficients  ranged  from  .34  to  .90  (mean  of  .60). 

An  oversight  occurred  in  administering  foe  454X1  JKT  in  the  current  study.  Because  of  foe  large 
number  of  test  items  (159),  the  original  JKT  was  divided  into  two  booklets  (labeled  A  and  B)  covering 
the  26  WTPT  tasks  in  the  Aerospace  Ground  Equipment  specialty-booklet  A  addressed  12  tasks  and 
booklet  B  addressed  14.  The  tasks  were  randomly  distributed  between  foe  two  booklets  to  allow  a  break 
between  testing  sessions.  In  the  present  study,  however,  booklet  A  was  inadvertently  omitted.  The 
reduction  in  task  coverage  would  be  expected  to  lessen  the  content  validity  of  foe  454X1  JKT  and  possibly 
the  reliability  of  the  job  knowledge  scores  obtained  on  airmen  in  this  specialty. 

Vocational  Interest  for  Career  Enhancement  Instrument 

The  Vocational  Interest  for  Career  Enhancement  (VOICE)  Instrument  (AFPT  600,  Form  D,  1  June 
1987)  was  selected  as  the  vocational  interest  inventory  for  use  in  this  study  because  it  was  specifically 
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designed  to  measure  the  job  interests  of  Air  Force  junior  enlisted  personnel  and  was  readily  available  to 
the  research  team  at  no  cost  Form  D  of  the  VOICE  is  a  160-item  interest  inventory  designed  to  measure 
an  individual’s  preference  for  different  types  of  work-related  activities.  It  consists  of  four  sections;  Jobs, 
Work  Tasks,  Spare  Time  Activities,  and  Desired  Learning  Experiences.  Under  each  section  are  listed  a 
series  of  brief  descriptions  or  statements  related  to  the  section  tide.  For  example,  under  Section  1:  Jobs 
are  listed  32  job  tides  (e.g.,  baker,  fanner,  plumber);  Section  3:  Snare  Time  Activities  includes  27 
activities  (e.g.,  go  canoeing,  listen  to  an  opera).  Individuals  indicate  their  preference  for  each  item  by 
marking  "L"  for  Like,  T  for  Indifferent,  or  "D"  for  Dislike. 

The  earliest  form  of  the  VOICE  contained  400  items  and  was  scored  on  18  Basic  Interest  Scales 
(BIS)  and  20  Occupational  Scales  (Alley,  Berberich,  &  Wilboum,  1977;  Alley,  Wilboum,  &  Betberich, 
1976).  The  BIS  measured  interest  in  different  areas  without  reference  to  particular  jobs,  whereas  the 
occupational  scales  expressed  the  level  of  job  satisfaction  examinees  would  be  expected  to  have  in 
particular  DoD  occupational  areas.  Because  of  the  large  number  of  scales,  VOICE  scoring  was  rather 
complex.  Later,  Watson,  Alley,  and  Southern  (1979)  developed  a  smaller  number  of  operational 
composites  to  reduce  the  number  of  occupational  scales,  and  also  to  simplify  scoring.  Their  factor  analysis 
of  the  occupational  scales  dictated  a  minimum  of  eight  composites. 

Fairbank  (1986)  conducted  research  which  led  to  the  development  of  Form  D  of  the  VOICE.  He 
identified  six  operational  composites  to  predict  job  satisfaction  in  the  20  DoD  occupational  areas.  These 
occupational  areas  encompassed  the  vast  majority  of  Air  Force  jobs.  The  magnitude  of  the  correlation 
between  the  six  composites  and  predicted  job  satisfaction  across  the  occupational  areas  ranged  from  .46 
to  .94.  Fairbank  (1986)  also  found  that  the  six  composites  could  be  used  in  place  of  the  BIS  without 
significant  loss  of  predictive  power.  Further,  the  composites  could  be  derived  directly  from  VOICE  item 
responses  without  first  calculating  BIS  scores. 

Alley  et  al.  (1976)  demonstrated  that  the  VOICE  is  related  to  reported  job  satisfaction.  In  the  current 
study,  it  is  hypothesized  that  incumbent  job  interest,  as  reflected  by  the  six  VOICE  composites,  may  also 
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relate  to  job  performance.  Existing  literature  on  the  relationship  between  job  satisfaction  and  job 
performance  is  inconclusive  (Seashore  &  Taber,  1975). 

Generalized  Motivation  Scale 


The  Generalized  Motivation  Scale  (GMS)  developed  by  Ringenbach  (1989a,  1989b)  was  selected  to 
measure  motivation  level  for  job  performance.  Several  considerations  guided  the  choice  of  the  GMS  for 
this  project  over  other  available  motivation  instruments.  Generality  of  application  was  a  key  reason. 
Because  the  GMS  is  general  in  nature,  rattier  than  tied  to  measurement  of  motivation  in  specific 
environments,  the  instrument  was  judged  to  be  suitable  for  assessing  motivation  to  demonstrate  drills  in 
the  different  jobs  studied  in  the  project  Further,  during  the  course  of  GMS  development  the  items  had 
been  reviewed  specifically  for  applicability  to  Air  Force  contexts.  Another  desirable  feature  concerned 
its  broad  coverage  of  the  construct  of  motivation.  Item  writers  considered  a  wide  range  of  attributes 
relevant  to  describing  a  motivated  worker,  including  behaviors  associated  with  such  critical  elements  as 
motives,  expectancies,  and  incentives.  Major  phases  of  the  motivational  process  (preparation,  execution, 
and  reaction)  were  also  addressed.  Readers  interested  in  a  detailed  description  of  the  theoretical 
framework  and  development  of  the  GMS  are  referred  to  Ringenbach  (1989b). 

The  GMS  instrument  consisted  of  30  statements  to  which  respondents  indicated  their  degree  of 
agreement  or  disagreement  using  a  7-point  Likert  scale.  Ringenbach  (1989b)  used  variable  reduction 
techniques  including  factor  analysis  to  select  the  30  items  from  a  larger  pool  of  102  experimental  items 
and  to  arrive  at  subscale  content  and  names.  Three  subscales  were  derived:  Persistence  in  the  Face  of 
Adversity,  Goals  and  Plans,  and  Evaluation  for  Competence.  The  number  of  items  in  the  subscales  were 
12, 10,  and  8,  respectively.  Subscale  scores  were  computed  by  summing  item  responses,  alter  reversing 
scale  values  for  negatively  phrased  items.  Ringenbach ’s  studies  with  samples  of  recruits  for  Basic  Military 
Training  showed  the  subscales  to  be  moderately  reliable.  Cronbach  alpha  coefficients  ranged  from  .63 
to  .83.  Construct  validity  was  examined  by  interconrelating  the  subscales  with  those  of  six  other 
motivation  instruments.  Results  were  mixed;  the  GMS  subscales  related  highly  with  those  for  some,  but 
not  all,  alternative  motivation  instruments. 
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A  copy  of  the  GMS  instrument  and  instructions  is  in  Appendix  E.  Note  that  the  won)  "Motivation" 
was  dropped  from  the  title  of  the  instrument  administered  to  airmen  in  this  project  The  reason  was  to 
control  potential  social  desirability  bias  by  discouraging  faked  responses. 

Task  Timing  Forms 

A  key  objective  of  this  study  was  to  evaluate  the  ability  of  supervisors  to  estimate  task  performance 
times  for  their  subordinates.  To  accomplish  this  part  of  the  study  it  was  necessary  to  obtain  actual  times 
on  a  subsample  of  the  airmen  performing  some  of  the  tasks  for  which  estimated  times  were  also  collected. 
The  number  of  tasks  and  airmen  to  be  timed  was  limited  by  two  factors.  First  die  project  schedule 
allowed  only  one  day  per  AFS  per  base  for  timing.  Second,  to  minimize  duty  section  disruption  and 
reduce  fatigue  effects,  a  goal  of  no  more  than  two  hours  of  timing  per  airman  was  established.  Within 
each  AFS  only  the  WITT  tasks  were  considered  for  timing  to  take  advantage  of  the  detailed  instructions, 
equipment  lists,  and  specific  task  steps  available.  In  two  of  the  four  AFSs  (454X1  and  455X2)  some  of 
the  WTPT  tasks  were  judged  unsuitable  for  timing  because  they  created  potential  safety  hazards,  were 
potentially  damaging  to  the  equipment  involved,  or  required  an  excessive  amount  of  time  to  perform.  The 
following  number  of  timing  tasks  were  selected  for  timing:  122X0  -11  tasks;  454X1  -  6  tasks;  455X2  - 
8  tasks;  and  732X0  -  8  tasks. 

A  cutoff  time  was  established  for  each  of  the  33  timing  tasks  to  standardize  measurement  procedures 
across  all  data  collection  sites  and  to  prevent  airmen  from  spending  too  much  time  on  any  one  task.  The 
concern  was  that  if  foe  tasks  were  left  open-ended,  some  less  proficient  airmen  might  continue  trying  to 
perform  a  particularly  difficult  task  at  the  expense  of  performing  a  variety  of  tasks.  The  use  of  cutoff 
times  was  preferred  over  more  subjective  stopping  rules  because  not  all  timing  sessions  were  conducted 
by  the  same  data  collection  personnel,  and  standardization  would  have  been  problematic.  The  cutoff  times 
were  set  at  the  slowest  benchmark  time  for  the  task,  plus  20  percent 

To  maximize  the  sample  sizes  for  task  level  analyses,  a  subset  of  die  timing  tasks  in  each  AFS  were 
designated  "core"  tasks  for  timing.  The  core  tasks  were  chosen  by  randomly  selecting  tasks  from  foe 
timing  ii$t  until  the  sum  of  their  cutoff  times  approached  90  minutes.  This  procedure  ensured  that  a 
common  set  of  tasks  in  each  AFS  was  performed  by  each  airman  during  the  2-hour  time  limit  After 
performing  the  core  tasks,  the  airmen  spent  die  remaining  time  performing  as  many  non-core  tasks  as 
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possible  to  maximize  the  number  of  timed  observations  on  each  airman.  This  was  done  to  maximize  the 
potential  reliability  of  productive  capacity  measures  derived  from  actual  time  measurements.  Table  4  lists 
all  of  the  tasks  in  each  AFS  selected  for  timing  and  identifies  which  were  designated  core  tasks. 


Table  4.  Tasks  Selected  for  Timing 


Aircrew  Life  Sap  port  Special*  (122X0) 

E199*  Make  entries  on  AFTO  Form  152  (chemical  ensemble  inspection  record) 

H295*  Fit  the  55/P  helmet  using  a  custom  liner 

H303  Perform  a  30-day  inspection  on  an  HGU-55/P  helmet 

H315  Replace  the  nape  strap  and  pad 

H320  Remove  and  replace  headsets  in  helmet 

1330  Size  end  fit  oxygen  masks 

1349  Perform  oxygen  mask  periodic  inspections 

J380*  Remove  and  install  the  filter  elements  in  the  CRU-80P 

J383*  Perform  the  mask  exchange  in  the  vapor  hazard  area 

K389  Fit  or  adjust  a  parachute  harness 

K398  Perform  30  day  routine  parachute  inspections 

AertMpace  Ground  Equipment  Special*  (454X1) 

F154*  Perform  an  aircraft  support  generator  service  inspection 
F155*  Perform  a  service  inspection  on  e  load  bank 
F162*  Perform  a  service  inspection  on  a  hydraulic  test  stand 
1284*  Remove  and  replace  an  alternator  belt 
1300*  Replace  the  flare  fitting  on  a  fuel  line 

PS49*  Perform  an  operator's  inspection  of  an  AF  vehicle,  AFTO  Form  373 

Avionic  Communication  and  Navigation  Syatoma  (455X2) 

E110  Research  and  identify  information  regarding  parts  using  technical  data 
E138*  Complete  AFTO  Form  349,  Maintenance  Data  Collection  Record 

F169  Visually  inspect  receiver-transmitter  (RT)  units  or  other  radioAest  equipsncsit  received  from  supply  or  manufacturers 

F199*  Safety  wire  aystem  components 

F200*  Property  set  up  a  flightline  maintenance  stand 

G218  Perform  a  minimum  standards  (bench)  check  on  a  ultra -high  frequency  (UHF)  receivcr-tranamitter 
G262*  Set  up  ultra -high  frequency  lest  equipment 
L398  Isolate  malfunctions  in  interphone  cords 

Person e!  Special*  (732X0) 

E35*  Draft  a  message  saying  that  a  job  inventory  was  mailed 

E102*  Construct  immediate  inquiries  to  determine  date  of  rank,  unit  (PAS),  and  duty  phone  for  someone  stationed  at  your  base 
E107*  Dispose  of  unclassified  PDS  products  of  only  a  few  pages 
EU4*  Sign  in  a  document  and  sign  out  again,  using  AF  Form  614 
P140*  Open  and  dose  a  CRT  in  the  CBPO 

L733*  Respond  to  a  request  to  release  information  from  a  member’s  fQe  in  accordance  with  the  Privacy  Act 
0826*  Properly  file  10  documents  in  personnel  records 
R1011*  Compute  service  dates 

*  Core  Tasks 
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The  timing  tasks  (both  core  and  non-core)  were  assembled  into  Tasks  for  Timing  booklets,  one  for 
each  AFS.  A  table  of  contents  listing  the  tasks  to  be  timed  was  included  on  the  first  page  of  each  booklet. 
Beginning  on  the  next  page  were  descriptions  of  die  tasks  to  be  timed.  The  task  descriptions  were 
identical  to  those  the  supervisors  used  in  the  time  estimation  workshops.  Each  task  was  headed  by  the 
task  number  and  title,  followed  by  the  benchmark  performance  times.  The  tools  and  equipment  needed 
for  completion  of  each  task  were  also  listed  so  they  could  be  set  up  before  the  timing  study  began.  A 
standard  background  statement  was  also  provided  for  each  task  to  answer  possible  questions  and  to 
maintain  consistency  between  bases,  commands  and  evaluators.  The  configuration  detailed  in  die  task 
setup  was  also  used  to  maintain  consistency  between  timings  of  each  task. 

Inside  the  back  cover  of  each  Task  for  Timing  booklet  was  a  5-point  performance  level  scale.  This 
scale  was  developed  to  assess  how  well  each  airman  performed  each  task  during  the  timing  sessions.  The 
definition  of  productive  capacity  used  in  this  study  required  that  tasks  not  only  be  performed  as  quickly 
as  possible,  but  also  be  performed  to  a  satisfactory  level  of  quality.  The  performance  level  scale  and 
description  is  shown  in  Table  5. 

Table  5.  Performance  Level  Descriptions 


Rating 

Performance  Levels 

Description 

5 

Exceptional 

Far  exceeds  the  acceptable  level  of  performance.  Completed  all  atepa,  critical  and 
noocritical,  efficiently  (skillfully)  and  accurately. 

4 

Distinguished 

Exceeds  the  acceptable  level  c f  performance.  Completed  all  critical  atepa  accurately, 
but  not  necessarily  efficiently.  Made  very  few  or  no  noocritical  errors. 

3 

Acceptable 

Meets  the  minimum  acceptable  level  at  performance.  Completed  all  critical  atepa 
accurately.  Made  some  noocritical  errors. 

2 

Deficient 

Below  the  acceptable  level  of  performance.  Made  same  critical  errors  or  numerous 
noocritical  errors. 

1 

Unacceptable 

Far  below  the  acceptable  level  at  performance.  Made  numerous  critical  and  noocritical 
errors  and/or  could  not  complete  the  task. 

In  addition  to  the  Tasks  for  Timing  booklets,  a  Task  Timing  Form  was  also  developed  for  each  AFS 
to  record  the  timing  results,  the  performance  level,  and  a  frequency  of  task  performance  measure.  The 
frequency  measure  used  an  R-O-N  scale  (Regularly,  Occasionally,  Never)  to  record  the  self-reported 
frequency  with  which  each  airman  performed  each  task  being  timed.  The  frequency  measure  was  collected 
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to  evaluate  the  potential  effect  of  direct  task  experience  on  performance  time.  A  sample  Task 
Timing  Form  (122X0  -  heading  and  first  task  only)  is  shown  in  Figue  3.  Copies  of  the  complete 
four  Task  Timing  Forms  (one  per  AFS)  are  contained  in  Appendix  F. 

TIMING  FORM  -  AIRCREW  LIFE  SUPPORT  (AFS  122X0) 

Examinee  Name  SSN 

UTt  Fust - Middle  Initial -  - 

Grade _  AFS _ _  Location  (AFB) _ 

Task  El$5  Make  entries  on  AFTO  Form  IS  (Chemical  ensemble  inspection  record 

RON 

lime: _ . _ . _  Cutoff:  8  mins.  24  secs.  Rating:  1  2  3  4  5 

Figure  3.  Sample  Task  Timing  Form 

Procedures 

Three  kinds  of  activities  occurred  during  field  Table  6.  Field  Data  Collection  Activity  Schedule 
data  collection.  These  were:  (1)  supervisor  task 
time  estimating  sessions,  (2)  airman  testing 
sessions,  and  (3)  task  timing  sessions.  Table  6 
summarizes  die  schedule  of  activities.  The  first 
day  of  each  field  visit,  which  was  always  Monday, 
was  used  to  conduct  two  supervisor  time 
estimation  sessions  and  two  airman  testing 
sessions.  Morning  and  afternoon  sessions  for  both 
supervisors  and  airmen  were  held  to  ensure  that 
all  selected  participants  had  ample  opportunity  to 
attend,  considering  their  work  schedules  and  other 
commitments.  The  two  different  kinds  of 
activities  were  held  concurrently,  normally  in 
rooms  that  were  in  close  proximity,  but  in  at  least 
one  case,  in  widely-separated  buildings.  The  remaining  four  days  of  the  field  visits  were  used  for  task 
timing  sessions,  one  day  for  each  AFS. 


D»; 

Activity 

Monday  morning 
(repeat  of  both 
«ea»ion»  indie 
afternoon) 

•  Supervisor  Time 

•  Airman  Testing  Session 

Tuesday 

•  Avionic  Comm-Nav 

Systems  (45SX2) 

Timing  Session 

Wednesday 

•  Aerospace  Ground 

Equipment  (454X1) 

Timing  Session 

Tbonday 

•  Aircrew  Life  Support 
(122X0)  Timing 

Session 

Friday 

•  Personnel  (732X0)  Tuning 

Session 
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Supervisor  Time  Estimating  Sessions 

Preparation  for  the  supervisor  time  estimating  sessions  began  before  the  opening  day  of  the  field  visit 
A  listing  of  all  airmen  and  supervisors  to  be  included  in  the  study  was  provided  by  the  base  being  visited. 
Before  die  first  day’s  activities,  the  Supervisor/Airman  Lists  were  prepared  for  each  supervisor,  listing 
each  of  their  subordinates  to  be  included  in  die  study,  with  the  Social  Security  number  for  each.  These 
lists  plus  blank  Background  Information  Forms,  Tasks  for  Time  Estimation  booklets,  and  one  blank  Time 
Estimation  Form  were  assembled  as  a  packet  to  be  handed  out  to  each  supervisor  as  they  entered  the 
testing  room.  The  Tasks  for  Time  Estimation  booklets  had  a  different  color  cover  for  each  AFS  to  help 
ensure  that  the  supervisors  used  the  correct  set  of  tasks. 

The  first  activity  of  the  session  was  to  ensure  that  airmen  had  been  properly  matched  to  the  correct 
supervisor.  Supervisors  were  asked  to  review  die  Supervisor/Airman  Lists  for  accuracy.  If  there  was  an 
airman  on  a  supervisor’s  list  that  was  not  his  or  her  subordinate,  the  supervisor  was  asked  to  identify  the 
airman’s  actual  supervisor,  if  they  were  present  in  the  room.  If  this  was  not  possible,  an  attempt  was 
made  to  locate  the  airman’s  supervisor,  then  later  in  the  week,  that  individual  was  contacted  and  asked 
to  rate  the  airman.  In  this  way,  every  airman  who  was  tested  also  received  task  performance  time 
estimates  from  his  or  her  supervisor. 

Supervisors  then  completed  a  Background  Information  Form  on  themselves  and  read  the  Privacy  Act 
statement  Next  they  were  asked  to  look  over  the  Tasks  for  Time  Estimation  booklets  and  to  assign 
performance  time  estimates  to  each  task  for  each  airman  on  their  list  The  test  administrator  read  very 
detailed  and  structured  instructions  to  the  supervisors  to  aid  them  in  assigning  task  performance  times 
(Appendix  G).  As  each  supervisor  finished  assigning  times  to  the  tasks  for  an  airman,  he  or  she  brought 
the  Time  Estimation  Form  to  the  test  administrator  and  was  given  another  blank  form.  This  procedure 
was  used  to  ensure  that  each  airman  was  rated  independently  on  each  task,  rather  than  having  each  task 
rated  across  all  airmen  assigned  to  the  supervisor.  This  process  continued  until  all  supervisors  had  rated 
all  of  their  airmen. 

Airman  Testing  Sessions 

A  color-coded  packet  of  materials  was  also  prepared  beforehand  for  each  airman  to  be  tested.  The 
materials  included  the  Background  Information  Form  (the  same  form  used  by  the  supervisors),  a  scannable 
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answer  sheet  (Appendix  H),  die  VOICE  instrument,  and  the  GMS  instrument.  Forms  and  instruments 
were  placed  inside  die  Job  Knowledge  Test  (JKT)  for  die  airman’s  AFS,  except  for  the  4SSX2  specialty 
(which  did  not  have  a  JKT).  These  packets  were  placed  on  the  desks  or  tables,  along  with  pencils,  before 
the  airmen  entered  the  testing  room.  As  airmen  entered  die  room,  they  were  asked  their  specialty,  and 
then  directed  to  a  desk  with  materials  of  the  appropriate  color.  The  color-coded  packets  also  enabled  test 
administrators  to  quickly  locate  and  keep  track  of  materials  used  during  the  week. 

At  the  time  designated  to  begin  the  session,  roll  was  called,  and  those  missing  or  those  present  but 
not  on  the  list  were  identified.  Airmen  who  did  not  appear  for  either  the  morning  or  afternoon  testing 
session  were  identified  and,  in  most  cases,  administered  the  tests  later  in  the  week.  Airmen  who  were 
present  but  whose  names  were  not  on  the  list  were  identified  and  in  some  cases,  determined  to  be 
substituting  for  others  not  present  In  any  case,  the  names  of  their  supervisors  were  noted,  so  that  later 
in  the  week  these  supervisors  could  estimate  task  performance  times  for  the  airmen. 

Airmen  were  then  given  instructions  for  completing  the  Background  Information  Form.  They 
completed  the  form,  read  the  Privacy  Act  statement  and  then  began  to  complete  the  tests.  The  test 
administrator  read  uniform  instructions  to  the  group  to  aid  diem  in  completing  the  instruments  properly. 
Copies  of  the  instructions  are  in  Appendix  I.  The  JKT  was  completed  last  so  that  airmen  from  the  455X2 
specialty  could  be  excused. 


Task  Timing  Sessions 

Prior  to  each  base  visit  a  point  of  contact  (POQ,  usually  a  senior  NCO,  was  identified  in  the  work 
center  for  each  AFS  to  help  coordinate  the  task  timing  sessions.  The  POCs  were  sent  a  list  of  the  tasks 
to  be  timed,  showing  the  equipment  and  materials  required  for  each.  On  Monday  afternoon,  members  of 
the  data  collection  team  visited  each  POC  to  discuss  any  problems  encountered  in  obtaining  the 
appropriate  equipment  and  to  identify  which  airmen  would  be  timed.  If  the  work  center  could  not  support 
timing  of  some  of  the  tasks  because  of  local  differences  in  equipment  or  procedures,  the  task  descriptions 
and/or  instructions  were  rewritten  to  describe  conditions  at  the  base.  Rewriting  tasks  in  the  field  to  reflect 
different  equipment  or  procedures  also  required  the  establishment  of  new  benchmark  performance  times. 
Six  NCOs  in  die  work  center  were  asked  to  independently  estimate  fastest,  normal  and  slowest 
performance  times  for  each  modified  task,  using  the  same  definitions  that  were  used  at  the  SME 
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workshops  at  which  the  benchmarks  for  the  original  tasks  were  established.  It  was  not  feasible  to  obtain 
group  consensus  on  the  revised  benchmarks  for  modified  tasks;  only  independent  estimates  were  collected. 
The  mean  of  the  independent  estimates  of  the  slowest  benchmark  times  were  used  to  revise  the  cutoff 
times  for  modified  tasks  (the  mean  times  were  increased  by  20  percent  to  set  the  cutoffs). 

Two  senior  NCOs  from  each  AFS  were  also  identified  during  the  Monday  afternoon  meetings  to 
assist  with  the  timing  process  and  to  evaluate  die  performance  level  on  each  task.  These  NCOs  were  7 
skill-level  personnel  or  above  who  were  familiar  with  the  work  center  operations  and  equipment  and  who 
had  hands-on  and  supervisory  experience  with  the  tasks  being  performed.  Care  was  taken  not  to  have  any 
of  the  senior  NCOs  evaluate  the  performance  of  any  of  their  direct  subordinates. 

At  the  beginning  of  each  day  (see  Table  6),  the  test  administrators  met  with  the  senior  NCOs  who 
would  be  assisting  with  the  timing  process  and  evaluating  performance  levels.  The  senior  NCOs  and 
airmen  to  be  timed  were  given  structured,  detailed  briefings  before  beginning  the  timing  trials.  Copies 
of  the  briefings  and  instructions  are  found  in  Appendix  J. 

To  maximize  the  number  of  observations,  there  were  generally  two  airman  being  timed  at  the  same 
time.  The  core  tasks  were  always  performed  first,  followed  by  as  many  non-core  tasks  as  could  be 
measured  in  the  remaining  time  (up  to  the  two  hour  limit).  To  the  extent  possible,  airman  performed  tasks 
in  a  random  sequence  (within  the  two  categories,  core  and  non-core)  to  control  anxiety  and  fatigue  effects 
that  might  occur  due  to  task  presentation  order.  At  times,  however,  the  equipment  needed  by  one  airman 
was  in  use  by  the  other  airman  being  timed;  in  these  cases,  the  next  task  in  the  random  order  was 
performed,  with  the  task  that  was  skipped  being  performed  as  soon  as  the  equipment  was  available. 

The  task  timing  process  began  with  the  airman  filling  out  the  identification  portion  of  a  Task  Timing 
Form  for  his  or  her  AFS.  The  administrator  (a  member  of  the  research  team)  then  selected  the  task  to  be 
timed  and  had  the  airman  indicate  how  frequently  he  or  she  had  performed  fire  task,  using  the  R-O-N  scale 
on  the  Task  Timing  Form.  The  senior  NCO  ensured  that  the  required  tools  and  equipment  were  available, 
and  that  the  work  area  was  configured  in  accordance  with  the  description  in  the  Tasks  for  Timing  booklet 
The  senior  NCO  then  reviewed  the  "instructions  to  the  administrator"  for  the  task  (contained  in  the 
booklet),  and  read  the  "say  to  the  incumbent"  section  of  the  task  description  which  told  the  airman  exactly 
what  task  was  to  be  performed.  As  the  airman  began  performing  the  task,  the  administrator  started  the 
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stopwatch.  Upon  completion  of  the  task,  the  stopwatch  was  stopped  and  the  elapsed  time  recorded  on  the 
Task  Timing  Form  by  the  administrator.  Using  the  task  steps  specified  in  the  Tasks  for  Timing  booklet 
as  a  guide,  die  senior  NCO  observed  the  airman’s  performance,  compared  it  with  the  five  performance 
levels  described  at  the  back  of  the  booklet,  and  recorded  their  judgment  of  quality  level  on  the  Task 
Timing  Form.  If  the  airman  could  not  complete  the  task  prior  to  the  cutoff  time,  he  or  she  was  stopped 
and  the  cutoff  time  was  recorded  on  the  Task  Timing  Form,  along  with  a  performance  level  of  "1" 
(Unacceptable).  The  administrator  then  selected  the  next  task  to  be  performed  and  accompanied  the 
airman  and  senior  NCO  to  the  appropriate  work  station,  where  the  timing  process  was  repeated. 

In  some  cases  these  procedures  had  to  be  modified  to  accommodate  unique  situations.  Specific 
timing  procedure  modifications  for  each  specialty  are  discussed  in  Appendix  K. 

Data  Base  Design 

An  important  objective  of  the  current  effort  was  to  provide  a  means  for  future  analyses  of  the 
measures  collected  during  the  field  test  To  that  end,  a  data  base  was  designed  and  developed  to  provide 
an  accurate  and  complete  record  of  information  obtained  from  participants  in  the  project  (SMEs,  airmen, 
supervisors,  and  evaluators).  The  data  base  was  designed  to  record  three  categories  of  information:  (1) 
airman  data,  (2)  supervisor  data,  and  (3)  task  data.  Table  7  identifies  the  source  (e.g..  Background 
Information  Form,  scarmable  answer  sheet)  and  variables  (e.g..  Time  in  Air  Force  in  years  and  months, 
MAJCOM  code)  recorded  for  airmen.  Tables  8  and  9  summarize  the  information  recorded  for  supervisors 
and  for  tasks. 

The  scannable  answer  sheets  used  to  collect  VOICE,  GMS  and  JKT  item  responses  were  optically 
scanned  into  the  data  base;  all  other  data  were  manually  entered  into  Lotus  1-2-3  spreadsheets  using  the 
Data  Addition  Verification  Editing  (DAVE)  attachment  The  DAVE  software  facilitated  data  entry  by 
allowing  customized  data  entry  screens  to  be  developed  which  corresponded  to  the  format  of  the  original 
data  collection  forms.  Further,  DAVE  provided  a  data  audit  capability;  boundary  conditions  were  set  to 
detect  and  correct  certain  key  stroke  errors. 

After  data  entry  using  DAVE,  the  data  on  each  form  were  verified  against  the  spreadsheet  entries  and 
were  corrected  where  necessary.  Verification  of  the  Background  Information  Forms  consisted  of  checking 
the  names  and  social  security  numbers  for  all  forms.  All  background  data  was  verified  for  three  locations 
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Table  7.  Airman  Data 


(Travis,  Beale,  George  AFBs);  the  number 
of  errors  on  those  forms  was  less  than 
0.01  percent,  therefore  a  random  sample 
of  5  percent  of  the  remaining  forms  were 
checked.  All  variables  on  all  of  the  Time 
Estimation  Foims,  Task  Timing  Forms,  as 
well  as  information  on  the  tasks,  were 
verified  after  data  entry  and  corrected 
where  necessary.  The  data  were  then 
converted  into  three  ASCII  data  bases, 
one  for  each  category. 


The  VOICE,  GMS  and  JKT  answer 
sheets  were  scanned  by  a  National 
Computer  Systems  OpScan  5  scanner. 
Edit  checks  were  programmed  into  the 


From  Background  bfonutka  Foret 

Name  Grade 


SSN 

Time  in  Air  Force 
AFSC 

Time  in  AFS 


Time  in  Grade 
Time  in  Current  Poeitioo 
St^jervitor  (yeiAto) 
Number  of  Subordinate! 


MAJCOM  Time  ai  Supervisor  in  AFS 

Location 


From  ScannaUe  Answer  Sheets 

Name  VOICE  responiea 

Location  GMS  responses 

SSN  JKT  responses 

AFSC 

Froa  Tlae  Eartastton  Foret 

SSN 

How  Often  You  Observe  Incumbent  Perform  Task 
Time  Estimate 

Estimated  Productive  Capacity  (calculated  variable) 

Froa  Tatk  Timing  Fores  (only  selected  airmen) 

SSN 

Frequency  You  Perform  Task 
Time  to  Perform  Task 
Performance  Level  Rating 
Order  Tasks  Were  Performed 
Airman  Cutoff  (yet/no) 

Actual  Productive  Capacity  (calculated  variable) 


Table  8.  Supervisor  Data 


scanning  software  to  ensure  that  the 
location,  social  security  number,  and  first 
three  characters  of  the  AFSC  were 
complete  on  all  answer  sheets.  Any 
answer  sheet  with  one  or  more  of  these 
variables  missing  had  to  be  corrected  by 
hand  before  the  scanner  would  read  and 
store  the  data.  Another  edit  check  was 
designed  to  flag  answer  sheets  containing 
multiple  responses  by  airmen  to  any 

individual  GMS,  VOICE,  or  JKT  item.  Each  item  flagged  by  the  scanning  software  was  checked  against 


From  Background  Information  Forum 
Name  Grade 

SSN  Time  in  Grade 

Time  in  Air  Force  Time  in  Current  Position 

AFSC  Supervisor  (yesMo) 

Tone  m  AFS  Number  of  Subordinates 

MAJCOM  Time  as  Supervisor  in  AFS 

Location 


From  So  pei  vtsur/ Airman  Lists 

Names  of  Airmen  Rated  by  Supervisor 
SSNs  of  Airmen  Rated  by  Supervisor 
Tone  Supervised  Airmen  (in  months) 


the  answer  sheet,  and  corrected  if  possible.  Otherwise,  the  multiple  response  entries  were  recoded  as 
invalid,  using  an  asterisk  (*). 


Several  calculated  variables  were  added  to  the  airman  data  base,  including  die  one  of  primary  interest 
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Table  9.  Task  Data 


to  this  study,  productive  capacity  (PC). 

There  were  actually  two  PC  variables  - 
estimated  PC  and  actual  PC  Estimated 
PC  was  calculated  for  each  task  given  a 
time  estimate  by  supervisors,  and  actual 
PC  was  calculated  for  tasks  on  which 
airmen  were  timed.  In  addition  to  PC 
values,  other  calculated  variables  included 
(for  each  airman)  foe  six  occupational  interest  composites  derived  from  foe  160  VOICE  item  responses, 
the  force  motivation  sub-scales  derived  from  the  30  GMS  item  responses,  and  an  aggregate  job  knowledge 
score  (percent  correct)  derived  from  foe  JKT  item  rcsponses-each  of  foe  three  JKTs  had  a  different 
number  of  items  (122X0  had  63  items,  454X1  had  73,  and  732X0  had  47). 

The  task  data  base  (Table  9)  was  designed  to  provide  comprehensive  information  on  both  original 
tasks  and  modified  tasks.  A  task  numbering  system  was  developed  to  distinguish  foe  two  types  of  tasks. 
Further,  the  three  benchmark  times  (fastest,  normal,  and  slowest)  and  cutoff  time  (slowest  benchmark  plus 
20  percent)  were  recorded  for  each  task.  For  the  original  tasks,  foe  benchmarks  and  cutoff  timw  were 
obtained  from  information  collected  during  foe  SME  workshops.  For  the  modified  tasks,  foe  data  entered 
were  based  on  estimates  of  benchmark  times  obtained  from  senior  NCOs  during  the  field  test  Additional 
variables  were  created  to  provide  a  detailed  record  of  foe  types  of  modifications  to  foe  timed  tasks 
(equipment,  instructions,  benchmarks)  and  foe  location  (base)  at  which  specific  modified  tasks  were  timed. 
Table  10  shows  foe  tasks  that  were  modified  for  each  AFS,  how  foe  tasks  were  modified,  and  foe  bases 
where  the  modified  tasks  were  timed.  Note  that  the  data  base  was  designed  to  identify  tasks  with  multiple 
modifications.  For  example,  task  G262  (453X2)  was  modified  twice  (each  modification  affecting 
equipment,  instructions,  and  benchmarks).  The  first  modification  took  place  at  Beale  AFB  and  was 
subsequently  used  at  George  AFB;  foe  second  modification  occurred  at  Davis-Monthan  AFB.  The  original 
benchmarks  and  modified  benchmarks  were  recorded  in  the  task  data  base. 


SME  Wartuhor*  (awl  Field  VWts) 
Talk  Number! 

Fasten  Benchmark  Time 
Normal  Benchmark  Time 
Slowest  Benchmark  Time 
Cot-off  Time 

Field  VMts 

Modified  (yei/ho) 

TVpe  of  Modification  Of  modified) 
Where  Modified  Of  modified) 
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Table  10.  Task  Modifications 


Specialty 

Taak 

Type  of  Modification 

Locattoa(a) 

122X0 

El  99 

Equipment,  Inttnictian  end 
Benchmark 

Geoff e,  Holloman,  Davit -Monthan  and  Langley 

1330 

Imtractioo  end  Benchmark 

Ofiutt 

1349 

Equipment  end  Bendunuk 

Tiavit 

454X1 

F162 

Equipment 

Davit-Monthan 

1284 

Equipment 

Geoff  e,  Davit-Monthan  and  Ofiutt 

B 

Equipment,  Inttnictian  end 
Benchmark 

Holloman  and  Eglin 

455X2 

E110 

Equipment 

Beale  and  Davia-Monthan 

P169 

Equipment  end  Benchmark 

Beak,  Geotf e,  Shaw, 

Davit-Monthan,  Ofiutt  and  Eglin 

F199 

Equipment 

Beak 

F199 

Equipment,  Ingtroctioo 
and  Beocfamaik 

Ofiutt 

G262 

Equipment,  Inttnictian 
end  Benchmark 

Beak,  Geutg e  md  Davit-Monthan 

G262 

Inttnictian 

Shaw 

L398 

Equipment 

Trivia  and  Langley 

L398 

Inttnictian  end  Benchmark 

Beak.  Geoffe.  Holloman. 

Davit-Monthan.  Langley. 

Shaw,  Ofiutt  and  Eglin 

732X0 

E35 

Equipment 

All  Batet 

E102 

Equipment,  Inttnictian, 
and  Benchmark 

All  Baaea 

ffl.  COMMENTARY  ON  METHODOLOGICAL  ISSUES 

This  section  provides  a  discussion  of  methodological  issues  relating  to  the  procedures  and  instruments 
used  in  this  project  The  commentary  will  be  from  the  perspective  of  how  they  might  function  in  an  Air 
Force-wide  operational  productive  capacity  data  collection  effort-as  such,  die  procedures  and  instruments 
used  in  the  project  for  purely  construct  validation  and  research  purposes  (i.e.,  the  VOICE,  GMS  and  JKT 
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instruments  and  the  hands-on  task  timing  forms  and  procedures)  will  not  be  addressed  until  the  section 
on  recommendations  for  further  research.  The  commentary  addresses  the  following  issues:  (1)  AFS  and 
task  selection,  (2)  benchmark  procedures,  and  (3)  the  time  estimation  process.  Each  section  will  include 
a  description  of  what  was  done  in  this  study,  a  discussion  of  the  feasibility  of  using  the  procedure 
operationally,  and  an  identification  of  concerns  and  issues. 

AFS  and  Task  Selection 


The  selection  of  AFSs  and  tasks  to  be  included  in  this  study  was  highly  constrained  by  foe  desire  to 
remain  consistent  with  prior  job  performance  measurement  data  collection  efforts.  The  selection  of  AFSs 
was  limited  to  those  previously  studied  under  the  JPMS  project,  of  which  there  were  only  eight  (two  from 
each  aptitude  area).  The  added  constraint  that  one  AFS  be  chosen  from  each  aptitude  area  reduced  the 
selection  process  to  a  series  of  four  binary  choices.  If  AFSs  were  being  selected  to  generate  a 
comprehensive  productive  capacity  data  base  from  which  global  relationships  could  be  derived  between 
PC  and  aptitude  and  experience,  obviously  different  criteria  would  have  to  be  used.  Since  it  is  unlikely 
that  measurement  in  every  AFS  will  ever  be  possible,  techniques  for  generalizing  relationships  from  a 
sample  of  AFSs  must  be  developed.  Some  suggestions  for  generalization  techniques  will  be  included  in 
the  final  section  of  this  report  The  generalization  technique  used  would  largely  dictate  the  AFS  selection 
strategy. 

Task  selection  within  AFSs  for  this  study  was  also  heavily  influenced  by  prior  JPMS  work.  To  take 
advantage  of  the  detailed  task  analysis  and  definition  previously  accomplished,  the  initial  selection  of  tasks 
was  limited  to  those  for  which  WTPTs  existed.  In  a  much  broader-based  data  collection  effort,  this 
restriction  would  not  apply,  with  all  tasks  in  the  master  task  inventory  being  candidates  for  selection. 
Again,  the  approach  selected  to  generalize  PC  relationships  from  a  sample  to  all  AFSs  would  also  dictate 
the  task  selection  methodology  to  be  used. 

We  found  great  variability  in  the  degree  to  which  tasks  were  standardized  across  commands  and 
bases-something  that  is  not  obvious  from  data  in  the  master  task  inventory,  since  all  of  foe  tasks  selected 
for  this  study  came  from  the  top  of  the  "percent  members  performing"  list  Task  standardization  was 
generally  a  function  of  the  equipment  being  used--obviously  the  standardization  of  equipment  drives  the 
standardization  of  the  tasks.  While  foe  task  information  routinely  collected  by  foe  Air  Force  (frequency 
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of  performance,  percent  time  spent  performing,  task  learning  difficulty,  etc.)  can  reasonably  be  estimated 
with  the  rather  generic  equipment  descriptions  available  in  the  task  inventory,  estimates  of  task 
nerfnnnance  time  would  be  expected  to  be  very  sensitive  to  the  specific  equipment  configuration  used. 
However,  choosing  to  gather  data  only  on  homogeneous  tasks  may  not  provide  an  adequate  productive 
capacity  data  base,  since  airmen  in  most  AFSs  are  rather  specialized  with  regard  to  the  tasks  they  do. 

Selecting  tasks  for  measurement  based  only  on  frequency  or  commonality  of  performance  also  ignores 
the  relative  importance  of  tasks  to  overall  mission  success.  The  formulation  for  PC  presented  earlier  in 
this  study  could  be  modified  to  include  a  factor  for  weighting  task-level  PC  indices  into  an  overall  index 
for  an  individual.  A  methodology  would  be  needed  for  establishing  the  weights  to  ensure  that  individuals 
are  selected  to  optimize  their  productivity  in  the  right  tasks,  not  Just  die  most  common  tasks.  Large 
increases  in  productivity  on  tasks  that  are  not  on  die  "critical  path"  to  mission  accomplishment  may  not 
improve  readiness  or  combat  capability  as  much  as  smaller  improvements  in  productivity  on  critical  tasks. 

Another  major  issue  with  task  selection  is  that  tasks  used  fortune  estimation  must  be  fully  developed, 
described  and  articulated  before  SMEs  can  assign  benchmark  times  or  supervisors  can  estimate 
performance  times  for  their  subordinates.  This  would  be  especially  true  if  the  time  estimates  were 
obtained  via  a  mail-out  survey,  where  no  opportunity  exists  to  discuss  and  standardize  the  task 
descriptions.  Fully-developed  tasks  must  include  information  about  exactly  what  equipment  is  bong  used 
and  what  state  it  is  in  at  die  beginning  of  the  task,  at  what  point  task  performance  begins  and  ends,  and 
some  idea  of  the  criterion  used  for  acceptable  performance.  During  the  discussions  of  the  tasks  at  the 
benchmark  estimation  workshops,  each  SME  had  his  or  her  own  idea  about  each  of  these  aspects  of  task 
performance.  The  tasks,  as  they  appear  in  the  master  task  inventories  from  OMS,  are  clearly  not  suitable 
for  use  in  estimating  performance  times  without  some  additional  description. 

Benchmark  Development  Procedures 

While  the  procedures  used  in  the  benchmark  time  estimation  workshops  worked  well,  the  cost  of  such 
workshops  to  routinely  establish  benchmark  times  for  large  numbers  of  AFSs  and  tasks  would  dearly  be 
prohibitive.  The  issue  discussed  above  relating  to  task  description  caused  most  of  the  difficulty  in  the 
workshops.  If  future  workshops  are  used  to  establish  benchmark  times,  the  SMEs  should  be  given  a  set 
of  fully-developed  tasks  with  which  to  start  time  estimation  and  discussion.  After  discussion  of  the  tasks. 
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so  that  every  person  is  clear  about  the  meaning  of  the  constraints,  the  SMEs  should  be  more  capable  of 
assigning  accurate  time  estimates. 

Although  extremely  time  consuming,  the  modified  NGT  process  worked  well  with  the  groups  of 
SMEs,  always  producing  consensus  among  the  members.  The  process  is  slow  because  of  the  large  number 
of  tasks  involved  and  the  requirement  for  three  separate  benchmark  times  for  each  task  (fastest,  normal 
and  slowest  performance  times),  which  necessitated  hundreds  of  iterations  through  the  process  of 
presenting  and  explaining  each  of  the  independent  estimates,  revising  the  estimates  and  finally  voting  on 
the  most  frequent  values  to  reach  consensus. 

A  word  of  caution  is  in  order  if  workshops  like  these  are  used  in  the  future.  In  the  interest  of 
expediency,  die  SMEs  for  this  effort  were  selected  from  only  a  few  bases,  generally  in  the  San  Antonio. 
Texas  area.  Unfortunately,  since  Air  Force  NCOS  tend  to  be  very  competitive,  this  appeared  to  have 
created  an  environment  in  which  SMEs  from  the  same  base  seemed  to  informally  conspire  to  estimate 
lower  performance  times  to  make  personnel  id  their  base  appear  more  proficient  than  at  the  other  bases. 
A  wider  sampling  of  bases  for  SME  selection  (preferably  with  only  one  SME  per  base)  might  reduce  the 
competitive  tendency.  Of  course,  if  future  benchmarks  are  established  via  a  mail-out  survey,  this  problem 
would  be  avoided. 

Time  Estimation  Process 

The  ability  of  supervisors  to  accurately  estimate  their  subordinates’  performance  times  on  specific 
tasks  will  also  be  a  major  analytic  issue  with  the  data  collected  in  this  study.  The  effect  of  the  level  of 
task  description  detail  on  estimation  accuracy  can  also  be  determined  through  analysis.  Procedurally,  the 
approach  used  to  solicit  supervisor  estimates  in  this  study  worked  quite  well,  although  it  would  not  be 
practical  for  any  large-scale  application.  Setting  up  group  supervisor  time  estimating  sessions  at  each  base 
was  a  non-trivial  task  One  difficult  problem  involved  getting  accurate  and  current  lists  of  supervisors  and 
their  subordinates,  since  this  information  changes  frequently  and  only  exists  at  base  level  (it  is  not  in  the 
master  files  at  AFMPQ. 

Another  practical  problem  involved  assembling  all  supervisors  in  one  location  at  the  same  time.  Shift 
work,  leave  and  TDY  (temporary  duty)  schedules,  mission  requirements,  and  availability  of  suitable  space 
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all  made  scheduling  the  group  sessions  problematic.  Again,  the  process  could  be  greatly  simplified  using 
a  mail-out  survey  approach  to  solicit  supervisor  estimates,  although  standardizing  the  estimation  process 
would  be  more  difficult  in  a  survey  mode;  for  instance,  it  would  not  be  possible  to  ensure  that  supervisors 
rated  each  subordinate  against  all  tasks  before  moving  on  to  the  next  subordinate,  as  was  done  in  the 
group  sessions.  The  need  to  match  supervisors  and  subordinates  ahead  of  time  in  this  study  was  driven 
by  the  research  nature  of  the  effort  (i.e.,  administration  of  VOICE,  GMS,  JKTs  and  hands-on  timing  tests 
to  the  subordinates  to  validate  the  supervisors’  estimates).  In  an  operational  system  supervisors  could 
simply  be  sent  a  task  booklet  and  time  estimation  forms  and  be  asked  to  identify  (name  and  SSAN)  all 
of  their  current  subordinates  and  provide  time  estimates  for  each. 

Another  procedural  issue  came  up  early  in  the  study,  relating  to  the  way  in  which  benchmark  times 
are  presented  to  supervisors  on  the  data  collection  instruments.  The  initial  strategy  was  to  just  include 
the  times  (fastest,  normal  and  slowest)  from  the  SME  workshops  in  the  Tasks  for  Time  Estimation 
booklets  presented  to  the  supervisors.  However,  after  preparing  the  booklets,  it  appeared  that  the 
benchmark  times  might  get  "lost"  in  the  descriptive  material  and  might  not  serve  their  intended  purpose 
of  keeping  the  time  estimates  within  reasonable  bounds.  To  correct  this  deficiency,  the  benchmark  times 
were  printed  on  the  Time  Estimation  Forms  for  the  pilot  test  at  Andrews  AFB.  Unfortunately,  some 
supervisors  interpreted  foe  three  benchmark  times  as  the  only  allowable  responses  for  each  task,  like  a 
multiple-choice  test  Several  scales  and  graphic  depictions  were  developed  to  reach  a  compromise 
between  these  two  approaches.  The  open-ended  time-line  approach  (see  Figure  1)  seemed  to  work  the 
best 


IV.  RECOMMENDATIONS 


Initial  Analysis  Needs 

The  data  collected  in  this  study  were  intended  to  help  answer  a  wide  variety  of  questions  related  to 
the  measurement  of  productive  capacity  and  its  suitability  as  an  Air  Force  personnel  planning  and 
management  tool.  The  data  base  produced  from  the  research  is  extremely  rich  and  complex  and  should 
support  many  future  analytical  studies.  The  following  are  some  of  foe  key  questions  that  should  be 
answered  before  proceeding  with  any  additional  data  collection  or  procedural  development  efforts: 
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Is  there  variability  in  PC  across  airmen?  Air  Force  enlisted  personnel  are  selected,  classified  and 
trained  through  a  rigorous  process  of  highly  standardized  testing  and  instruction.  A  key  question  that  must 
be  answered  before  proceeding  with  this  effort  is  how  much  variability  in  task  performance  times  (and, 
thus,  PC)  remain  among  individuals  who  have  been  through  this  "leveling"  process.  If  all  airmen  in  an 
AFS  perform  the  same  tasks  at  essentially  the  same  rate,  PC  will  have  little  value  as  a  discriminator.  That 
is,  low  variability  in  actual  task  performance  times  would  suggest  that  PC,  as  measured  in  the  current 
project,  may  not  be  a  useful  construct  for  personnel  planning  and  management 

Is  PC  a  valid  measure  of  iob  performance?  While  PC  has  an  intuitive  appeal  as  a  useful  performance 
measure,  its  relationship  to  other,  more  established,  performance  measures  has  yet  to  be  determined.  This 
fundamental  question  can  be  addressed  by  examining  the  relationships  between  mean  PC  values  computed 
for  each  airman  and  die  two  other  performance  measures  collected  on  each  airman,  i.e.,  the  overall 
productivity  rating  provided  by  the  airman's  supervisor,  and  the  JKT  score  achieved  by  die  airman  (AFSs 
122X0,  454X1  and  732X0  only).  This  analysis  should  be  conducted  using  both  estimated  task 
performance  times  and  actual  task  performance  times  to  calculate  mean  PC  values  for  the  airmen- 
obviously,  die  sample  sizes  will  be  much  smaller  for  die  actual  task  performance  time  case.  Correlational 
analyses  of  the  measures  will  help  establish  die  construct  validity  of  PC  as  a  performance  metric. 

How  well  can  supervisors  estimate  task  performance  times  for  their  subordinates?  Even  if  there  were 
variability  in  PC  and  it  proved  to  be  a  valid  performance  construct,  its  utility  as  a  personnel  management 
tool  will  be  severely  limited  unless  there  were  an  inexpensive  way  to  measure  it  This  issue  can  be 
addressed  through  a  variety  of  analyses  using  the  data  collected  in  this  study.  At  the  most  fundamental 
level,  the  association  between  estimated  and  actual  performance  times  can  be  examined  at  the  task,  AFS 
and  total  sample  levels.  If  the  relationships  are  not  moderately  strong  and  in  the  expected  positive 
direction,  a  method  other  than  supervisor  estimation  may  have  to  be  found  to  collect  PC  data.  Additional 
analyses  should  control  for  potential  effects  due  to  foe  frequency  with  which  supervisors  had  previously 
observed  airmen  performing  the  tasks,  using  responses  to  the  frequency  scale  (regularly,  occasionally, 
never)  on  foe  Time  Estimation  Form.  Controlling  for  supervisor  experience  (from  foe  background 
information  collected  on  foe  supervisors)  could  also  help  develop  supervisor  selection  criteria  for  future 
studies  (e.g„  do  more  experienced  supervisors  make  better  estimates?).  Higher  level  analyses  can  also  be 
accomplished  using  calculated  mean  PC  values,  determined  from  both  estimated  and  actual  performance 
times. 
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How  well  can  SMEs  set  benchmark  performance  times?  To  help  supervisors  estimate  task 
performance  times  for  their  subordinates,  drey  need  some  prior  information  regarding  the  range  of  times 
typically  required  for  each  task.  In  this  study  the  benchmark  times  were  generated  at  SME  workshops. 
Analyses  can  be  conducted  at  the  AFS  and  total  sample  level  comparing  the  benchmark  fastest,  normal 
and  slowest  times  with  the  actual  observed  minimum,  mean  and  maximum  performance  times  for  each 
task  on  which  time  measurements  were  taken.  This  analysis  will  help  indicate  whether  SME-generated 
benchmark  times  are  accurate  enough  to  guide  supervisors  in  estimating  performance  times  for  their 
subordinates.  The  analysis  of  fastest  time  benchmarks  (versus  minimum  measured  times)  will  be 
especially  critical,  since  the  fastest  time  benchmarks  are  proposed  for  use  as  the  T*  value  in  computing 
PC  values.  If  results  indicate  that  some  observed  or  supervisor  estimated  times  are  smaller  than  the 
benchmark  fastest  times,  using  the  benchmarks  to  compute  PC  values  may  have  to  be  reevaluated. 

Is  PC  predictable  from  available  personnel  information?  Even  if  PC  did  vary  across  airmen,  was  a 
valid  measure  of  job  performance,  and  was  measurable  in  some  cost-effective  manner,  it  would  not  be 
useful  for  selection,  classification  or  retention  planning  and  management  unless  it  was  predictable  from 
some  set  of  individual  and  job  characteristics.  Modeling  the  relationship  between  PC  and  individual 
characteristics  such  as  aptitude  (ASVAB  sub-test  scores  or  composites  from  die  personnel  files), 
experience  (from  the  background  information  collected  on  each  airman),  motivation  (from  the  GMS), 
interest  (from  VOICE),  and  job  characteristics  (such  as  aptitude  area)  would  begin  establishing  die  basis 
for  using  PC  as  a  selection,  classification  and  force  management  tool. 

Alternative  Strategies  for  Productive  Capacity  Measurement 

If  the  analyses  identified  above  confirm  that  PC,  using  supervisor  estimates  of  task  performance  time, 
has  potential  as  a  criterion  for  personnel  planning  and  management,  the  next  step  in  die  research  process 
should  be  aimed  at  moving  the  technology  closer  to  an  operational  system.  A  logical  move  in  this 
direction  would  be  an  attempt  to  collect  supervisor  estimates  of  task  performance  times  using  a  mail-out 
survey  rather  than  group  sessions.  In  this  way,  much  larger  samples  could  be  obtained  at  considerably 
lower  costs.  The  issue  relating  to  die  importance  of  benchmark  times  in  obtaining  accurate  and  reliable 
supervisor  estimates  could  be  addressed  by  including  benchmarks  on  some  of  the  surveys  and  not  on 
others.  The  issue  of  how  much  task  description  detail  is  needed  could  also  be  addressed  by  varying  the 
amount  of  detail  provided  to  die  supervisors. 
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Prior  to  developing  the  survey,  tasks  would  have  to  be  selected,  described  in  detail  and  benchmark 
times  established.  Ratlier  than  repeating  the  procedure  used  in  this  study  of  first  selecting  representative 
AFSs  and  then  selecting  representative  tasks  from  each  AFS,  an  alternative  approach  might  be  considered 
that  could  facilitate  the  generalization  process,  fit  a  recent  study,  Lance  and  Mayfield  (1989)  developed 
and  validated  a  26  item  standard  Air  Force  enlisted  specialty  task  taxonomy.  If  tasks  were  selected  and 
data  collected  by  task  category,  cutting  across  a  variety  of  AFSs,  it  might  be  possible  to  model  the 
relationship  between  PC  and  individual  characteristics  for  each  of  the  26  categories.  Weighted 
combinations  of  the  task  category  relationships  could  then  be  used  to  create  PC  relationships  for  each 
AFS,  with  die  weights  derived  from  an  analysis  of  the  relative  time  spent  (RTS)  on  each  of  the  task 
categories  by  airmen  in  each  AFS. 

In  an  operational  system,  the  RTS  distributions  would  have  to  be  periodically  updated  and  revalidated 
(possibly  as  part  of  the  routine  OMS  survey  process)  as  AFSs  change  over  time.  An  added  benefit  of  this 
approach  is  that  productivity  curves  could  also  be  estimated  for  new  weapon-system  driven  AFSs  by 
estimating  the  RTS  distributions  for  the  new  AFSs  to  weight  the  26  task  category  productivity  functions. 

Once  tasks  are  selected  for  analysis  from  the  master  job  inventories,  some  process  would  have  to  be 
used  to  describe  them  in  more  detail.  This  would  probably  require  convening  small  groups  of  SMEs  (3 
or  4)  for  a  few  hours  to  review  the  tasks  and  provide  a  set  of  conditions,  assumptions  and  equipment 
configurations  for  each.  The  fully  described  tasks  could  then  be  sent  out  to  a  larger  group  of  SMEs  with 
instructions  for  estimating  benchmark  performance  times-the  independent  estimates  could  then  be 
averaged  to  arrive  at  the  benchmarks.  The  increased  use  of  surveys  to  collect  performance  time  estimates 
will  require  careful  attention  to  the  instructions  provided  and  the  format  of  die  materials  sent  out  to 
minimize  misunderstandings  and  errors. 

Implications  for  the  Use  of  Productive  Capacity  in  Personnel  Planning 

This  study  clearly  demonstrated  that  collecting  PC  information  on  a  large-scale  basis  is  an  extremely 
complex  and  time-consuming  task.  Numerous  administrative  and  logistical  problems  were  encountered 
in  selecting  and  preparing  tasks  for  analysis,  in  establishing  benchmark  performance  times  for  each  task, 
in  identifying  and  assembling  supervisors  and  airmen  for  data  collection,  and  especially  in  timing  actual 
task  performance  in  an  operational  environment  However,  a  primary  purpose  of  die  effort  was  to  identify 
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these  problems  so  they  might  be  avoided  in  future  work.  Also,  a  secondary  purpose  was  to  collect  extra 
data  (interest  and  motivation  measures,  job  knowledge  scores  and  actual  performance  times)  to  validate 
the  use  of  PC  in  personnel  management,  something  that  would  not  be  required  in  routine  operational  use. 


In  spite  of  the  complexity  and  difficulties  encountered,  nothing  was  found  from  a  procedural  or 
instrumentation  standpoint  that  would  preclude  the  development  and  use  of  a  productivity-based  personnel 
planning  and  management  system  for  the  Air  Force.  The  development  of  such  a  system  is,  of  course, 
contingent  on  positive  results  from  the  analyses  identified  above  and  on  successful  follow-on  research  to 
refine  the  data  collection  procedures.  Analytical  techniques  must  also  be  developed  and  validated  for 
using  PC  relationships  to  set  minimum  enlistment  and  classification  standards,  to  allocate  new  recruits  to 
jobs,  and  to  set  reenlistment  objectives  that  will  yield  foe  highest  level  of  productivity  per  personnel  dollar 
spent 
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Supervisor-Airman  List 
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SUPERVISOR-AIRMEN  LIST 

Supervisor  Name _ 

Last 

SSN _ 

Grade _ 

AFS _ 

Location  (Beale  AFB,  etc.) 

Airmen 

Name  SSN 

(Last,  First,  Middle  Initial) 

L 

2. 

3. 

4. 

5. 

6. 

7. 

& 

9. 

10. 


Grade  How  many 
months  have 
you  supervised 
this  airman? 


First  Middle  Initial 
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APPENDIX  B 


Background  Information  Form 
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BACKGROUND  INFORMATION  FORM 


Name _ 

Last  First  Middle  Initial 

SSN _ 

Time  in  Air  Force _ years  _ months 

Air  Force  Specialty  (AFS)  Code  _____  Time  in 

MAJCOM  (TAC,  MAC,  etc.)  _______  Are  you 

Location  (Beale  AFB,  etc.) _ 

Grade _  Time  in  Grade _ years  _ .months 

Title  of  Current  Position _ 

Time  in  Current  Position _ years  _ months 

Are  you  a  supervisor?  yes _  no  _  If  yes,  number  of  subordinates _ 

Total  time  as  a  supervisor  in  this  AFS  years  _ months 


AFS _ years _ months 

in  the  Reserves?  yes _ no , 


PRIVACY  ACT  STATEMENT 

In  accordance  with  AFR  12-35  paragraph  8,  the  following  information  is  provided  as  required  by  the  Privacy  Act 
of  1974. 

AUTHORITY:  AFR  169-3,  Using  Human  Subjects  in  Research,  Development,  Test  and  Evaluation;  10  U.S.G  8012, 
Secretary  of  the  Air  Force,  Power  and  Duties,  Delegation  by  Executive  Order  9397,  22  November  1943. 

PURPOSE:  To  collect  information  from  incumbents  of  the  Aerospace  Ground  Equipment  (AGE)  (454X1), 
Personnel  (732X0),  Aircrew  Life  Support  (122X0),  and  Avionic  Communication  and  Navigation  Systems  (455X2) 
Specialties  regarding  task  performance.  Some  incumbents  will  complete  the  Vocational  Interests  for  Career 
Enhancement  (VOICE),  the  General  Scale,  and  a  Job  Knowledge  Test  (JKT). 

ROUTINE  USE:  The  information  to  be  collected  in  this  field  visit  supports  Air  Force  research  to  improve  its 
selection  and  classification  process.  The  information  is  for  research  purposes  only  and  will  not  become  part  of  any 
participant’s  permanent  personnel  record  nor  will  it  affect  any  participant’s  opportunity  for  promotion,  assignment 
or  retention. 

PARTICIPA*nON:  Your  participation  in  this  project  and  fomishing  of  your  Sodal  Security  Number  (SSN)  is 
voluntary.  However,  your  cooperation  in  this  effort  is  vitally  important  in  helping  to  ensure  that  future  Air  Force 
members  are  properly  selected,  trained  and  placed  in  the  right  job.  Your  SSN  is  essential  for  matching  information 
in  Air  Force  personnel  files  and  conducting  meaningful  analyses.  Failure  to  provide  it  could  render  your  other 
information  unusable. 
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APPENDIX  C 


Tasks  for  Time  Estimation  Booklet 
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Figure  CM.  Instructions  to  Supervisor  on  Task  Time  Estimation 

INSTRUCTIONS 

The  first  column  of  the  ume  estimation  form  is  a  list  of  task  numbers.  These  task  numbers  correspond  to 
the  task  numbers  in  your  task  booklets.  The  task  numbers  on  the  form  and  the  task  booklets  are  in  the 
same  order.  The  next  set  of  columns  asks  you  to  tell  us  how  often  you  have  observed  your  airmen  doing 
each  of  the  tasks.  You  provide  this  information  by  checking  the  box  above  the  "R\  *6*  or  “N*  for  each 
task.  *R*  means  you  regularly  see  him  (or  her)  do  the  task.  *0*  means  you  have  seen  him  do  the  task 
occasionally,  and  *N*  means  you  have  never  seen  him  do  the  task. 

The  next  section  presents  a  time  line  with  three  time  estimates.  'Fastest*  refers  to  the  fastest  this  task 
could  be  done  and  still  meet  the  accepted  level  of  performance.  "Normal"  refers  to  the  ordinary  time  the 
average  airman  needs  to  do  this  task  to  the  accepted  level  of  performance.  'Slowest*  refers  to  the  longest 
amount  of  time  that  could  be  allowed  for  performance  of  this  task  without  negative  consequences  to  the 
job.  These  times  were  provided  by  six  NCOs  from  your  career  field.  These  are  provided  for  your 
information  only  and  should  not  limit  the  estimates  you  for  your  airmen.  If  you  think  one  of  your 
airmen  could  do  a  task  faster  than  the  'fastest'  time  provided,  please  put  down  what  you  think.  If  you  think 
one  of  your  airmen  would  take  longer  than  the  'slowest*  time,  put  that  down  as  well  Or  if  you  estimate 
that  your  airman  may  perform  more  quickly  or  slowly  than  the  "normal*  time,  record  that  time.  You  are 
free  to  record  time  you  think  appropriate  for  your  airmen  on  each  task. 

The  last  column  on  this  form  is  where  you  record  your  estimate  of  how  long  each  of  your  airmen  would 
take  to  do  each  of  the  tasks.  Record  your  time  in  hours,  minutes,  and  seconds,  writing  in  the  appropriate 
number  in  the  boxes  above  'Hr*.  'Min',  and  'Sec*.  If  you  have  never  seen  one  of  your  airman  perform  a 
task,  please  give  your  best  estimate  of  how  long  it  would  take  him  or  her.  Base  your  estimate  on  your 
knowledge  of  his  or  her  performance  on  similar  tasks  in  your  specialty. 

As  an  example,  think  about  the  task  of  'Starting  a  Car.*  Assume  that  you  are  estimating  how  long  it  would 
take  an  airman  to  sun  a  motor  pool  car  and  that  the  airman  has  to  check  all  minors,  gauges,  and 
seartteering  wheel  positions.  Look  at  the  completed  time  estimation  example  below.  The  box  above  *R* 
was  checked  since  you  have  regularly  observed  the  person  perform  the  task.  The  time  estimates  provided 
to  you  are  30  seconds  for  the  fastest  time,  1  minute  for  the  normal  time,  and  2  minutes  for  the  slowest 
time.  You  think  about  the  particular  airman  you  are  rating  and  decide  that  he  ukes  a  little  longer  than 
the  normal  time,  but  not  as  long  as  the  slow  time  and  decide  to  give  him  1  minute  and  22  seconds  as  the 
time  estimate.  Therefore,  you  would  put  a  *0*  for  zero  hours  in  the  Hr  box,  a  *1*  for  1  minute  in  the  Min 
box.  and  a  *22*  for  22  seconds  in  the  Sec  box.  Are  there  any  questions? 


We  are  also  asking  you  to  give  an  overall  judgement  about  each  airman  regarding  the  relative  amount  of 
accepuble  work  they  can  perform  in  a  typical  day.  This  rating  scale  is  presented  on  the  last  page  of  your 
time  estimation  form.  The  question  is  worded: 

In  your  specialty,  consider  the  maximum  amount  of  acceptable  work  that  can  be  done  by  a  person 
in  a  typical  day  as  100  percent.  What  percent  of  the  maximum  could  the  person  you  are  currently 
rating  do  in  a  typical  day?  Write  your  estimate  in  the  box  below. 

Look  at  the  example  below.  Assume  that  100%  is  the  amount  of  work  that  can  be  done  and  1% 

is  the  least  amount  of  work.  If  you  believe  that  the  airman  you  are  rating  can  perform  75%  of  the 
maximum  amount  of  work,  write  75  in  the  box  provided."  Pweamot 


In  making  your  cmimata,  please  think  about  how  long  it  would  take  each  airman  to  do  each  task  if  be  or 
she  were  working  as  quickly  as  they  could,  while  «w»im»iwi«»g  satisfactory  performance.  The  key  words  here 
are  working  quickly  and  sa  us  factory  performance, 
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Figure  C-2.  Examples  of  WTPT  Tasks  (AFS  122X0,  Aircrew  Life  Support) 


Task  E199 

Make  entries  an  AFTO  Form  1S2  (chemical  ensemble  inspection  record) 

Consensus  Time  Estimates:  Fast  -  2  mins.  30  secs.  Normal  -  4  mins.  Slow  -  7  mins. 

Tools  and  Equipment:  T.O.  14P3-1-131,  T.O.  14P4-1-151,  pen,  pencil,  blank  preprinted 
AFTO  Form  152,  bag  containing  the  aircrew  chemical  ensemble  items,  tape,  completed 
1574. 

Background:  The  different  commands  use  different  equipment  so  the  equipment  listed 
on  the  AFTO  Form  152  should  be  assembled  prior  to  testing.  Another  reason  to  have 
the  items  gathered  before  testing  is  that  some  bases  wiU  not  have  individual  kits 
already  assembled. 

Configuration:  A  bag  containing  the  aircrew  ensemble  items  listed  on  the  AFTO  Form 
152,  and  the  completed  AFTO  Form  1574  for  this  bag.  Assume  the  bag  contains  one 
of  each  item  in  estimating  times  to  perform. 

Instructions  to  the  administrator:  Incumbents  may  not  be  familiar  with  all  the  items  in 
the  bag.  However,  there  is  a  space  for  each  item  on  the  AFTO  Form  152.  A  number 
of  the  spaces  on  the  152  will  not  be  able  to  be  filled  in  by  simply  inventorying  the  bag. 
The  bag  is  to  include  the  following  general  item:  tape.  It  wfll  be  necessary  for  you  to 
complete  the  1574  for  the  bag  prior  to  administering  this  item.  You  will  give  the 
AFTO  Form  1574  to  the  incumbent. 

SAY  TO  THE  INCUMBENT 

Maj  John  Smith  of  the  7th  ARS  just  came  in  off  alert  and  you  need  to  inventory  the 
chemical  ensemble  bag  and  complete  a  new  AFTO  Form  15Z  I  know  a  number  of  blank 
spaces  cannot  be  filled  in  by  inventorying  the  bag  but  fill  in  all  spaces  you  are  able  to 
complete.  Assume  that  the  "O"  ring  was  replaced  when  the  filter  was  replaced.  Here  is  the 
AFTO  Form  1574  for  this  bag.  This  task  was  developed  in  reference  to  T.O.  14P3-1-131 
and  T.O.  14P4-1-151.  Tell  me  when  you  are  ready  to  begin. 

Steps: 


1.  Fill  in  the  aircrew  member’s  name 

2.  Fill  in  the  filter  pack  serial  number  (will  vary,  information  on  filter  pack) 

3.  Fill  in  the  serial  number  for  the  mask 

4.  Fill  in  the  filter  lot  number  (will  vary,  information  on  AFTO  Form  1574  and/or 

filter  pack) 
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Task  E199  (continued) 

5.  Fill  in  the  date  of  installation  for  the  filter  (will  vary,  information  on  AFTO 
Form  1574  and/or  on  filter  pack) 

6.  Fill  in  the  date  the  nOH  ring  was  replaced  (same  date  as  the  filter  pack) 

7.  Fill  in  the  date  tape  replaced  (will  vary,  information  on  AFTO  Form  1574 
and/or  on  filter  pack) 

Fill  in  the  quantity  on  hand  (assume  time  to  count  one  of  each)  for: 

8.  Filter  pack  assembly 

9.  Extra  filter  elements 

10.  Suspension  assembly 

11.  Mask  CBO,  MBU-13/P 

11  Hood,  HGU-41/P 

13.  Drawers,  cotton 

14.  Undershirts,  cotton 

15.  Gloves,  cotton 

16.  Gloves,  butyl  or  neoprene 

17.  Undercoveralls 

18.  Footcovers;  plastic 

19.  Capes 

Fill  in  the  size  for 

20.  Drawers,  cotton  (will  vary) 

21.  Undershirts,  cotton  (will  vary) 

21  Undercoveralls  (will  vary) 
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Task  H295 

Fit  the  55/P  helmet  mmg  a  custom  Hn^r 

Consensus  Time  Estimates:  Fast  -  4  mins.  Nonna]  -  6  mins.  Slow  -  10  mins. 

lagls  apd  Equipment:  T.O.  14P3-4-151,  55/P  helmet  shell  (with  liner),  scissors, 
matches,  pencil,  earcups,  earcup  pads,  bayonet,  bayonet  spacer,  bayonet  receiver,  12/P 
or  5/P  oxygen  mask. 

Actual  fitting  of  the  helmet  depends  on  personal  preference  of  the 
wearer.  It  also  involves  actual  drilling  of  holes  in  the  helmet  shell. 

£gQfigVH?tj<?n:  HGU-55/P  helmet  shell  without  receivers  but  with  liner  and 
communication  system  installed,  and  with  nape  strap  untied. 

Instructions  to  the  Administrate  Administer  at  any  workstation.  A  person  needs  to 
be  available  for  this  task. 

SAY  TO  THE  INCUMBENT 

I  want  you  to  perform  the  step-by-step  procedures  that  are  followed  when  you  fit  the  55/P 
helmet.  The  helmet  liner  has  already  been  placed  in  the  shed.  Stop  immediately  prior  to 
drilling  the  holes.  This  task  was  developed  in  reference  to  T.O.  14P3-4-151.  Ted  me  when 
you  are  ready  to  begin. 

Steps: 

1.  Have  the  user  don  the  helmet 

2.  Have  user  rotate  the  earcups  so  that  the  ears  fit  into  and  are  completely 
surrounded  by  the  earseals 

3.  Lace  and  adjust  the  helmet  nape  strap  by  tightening  the  lacing  cord  and  securing 
it  with  a  square  knot 

4.  Insert  each  bayonet  into  helmet  receiver  to  the  fourth  locking  position 

5.  Place  a  spacer  underneath  each  helmet  receiver 

6.  Locate  each  helmet  receiver  no  closer  to  the  edge  of  the  helmet  shell  than  1/2 
inch 

7.  Trace  lightly  around  each  spacer  with  a  pencil 

8.  Mark  the  screw  holes  with  a  pencil  or  sharp  instrument 
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Figure  C-3.  Examples  of  Non- Wiki  Tasks  (AFS  122X0,  Aircrew  Life  Support) 

Task  E203 

Make  entries  on  AFTO  Forms  335  (Anti-g  suit  inspection  data) 

Consensus  Time  Estimates:  Fast  -  10  secs.  Normal  -  12  secs.  Slow  -  15  secs. 

Task  E206 

Make  entries  on  AFTO  Forms  338  (Survival  kit  record) 

Consensus  Time  Estimates:  Fast  -  2  mins.  Normal  •  3  mins.  Slow  -  4  mins. 

Assumptions:  Preprinted  form 

Transcribe  from  an  old  form  to  a  new  form 

Task  E209 

Make  entries  on  AFTO  Farms  391  (Parachute  log) 

Consensus  lime  Estimates:  Fast  •  3  mins.  Normal  -  4  mins.  Slow  -  5  mins. 

Assumptions:  This  form  is  not  preprinted 

Transcribing  from  an  old  form  to  a  new  form 

Task  E212 

Make  entries  on  AFTO  Forms  406  (Mesh  net  survival  test  inspection  record) 

Consensus  Time  Estimates:  Fast  -  1  min.  Normal  -  2  mins.  Slow  -  3  mins. 

Assumptions:  Preprinted  form 

Transcribe  from  an  old  form  to  a  new  form 

Task  E213 

Make  entries  of  AFTO  Forms  334  (Helmet  oxygen  mask  and  connector  inspection 
record) 

Consensus  Time  Estimates:  Fast  -  30  secs.  Normal  -  45  secs.  Slow  -  1  min. 

Assumptions:  Preprinted  form 

Transcribe  from  an  old  form  to  a  new  form 

Figure  C-3.  Examples  of  Nm-WTPT  Tasks  (AFS  122X0,  Aircrew  Life  Support) 
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appendix  d 

Time  Estimation  Forms 
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AIRCREW  LIFE  SUPPORT  SPECIALIST 

AFS  122X0 


Name 


Airman's  Name 


Airman's  SSN 


R  •  Regularly 
O  •  Occasionally 
N  -  Never 


How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest* 


Hr  ■  Hours 
Min  m  Minutes 
Sec -Seconds 

What  Is  Incumbent's 
Performance 
Time? 

(Fll  in  box  below) 


El  99 

H295 

H303 

H315 

H320 

L_ 

Ld 

Ld 

It 

'***“'  2  min  4  min  7  min  "" ""*■ 

30  sec 

Hr 

Min 

Sec 

LJ 

□ 

ip 

— 

— 

— 

111 

III 

t 

4  min  6  min  10  min  ^ 

~ 

Min 

Sec 

p"  ■  i 

* 

□ 

□ 

p 

□ 

□ 

t 

***^~~-  to  min  13  min -  17  min*"  ** 

~hT 

"muT 

Sec 

□ 

P 

p 

— 

Ej 

til 

t 

i  min  2  min  4  min 

45  sec 

Hr 

Min 

Sec 

r  * 

* 

□ 

□ 

p 

□ 

□ 

□ 

1  min  2  min  3  min  ^ 

~hT 

Min 

Sec 

1330 

□ 

□ 

□ 

— 

3 

H 

3  min  4  min  5  min  1  •*" 

30  sec 

Hr 

Min 

Sec 

1349 

□ 

□ 

II 

H 

N 

7  min  10  min  15  min  “ 

"hT 

"Min 

Sec 

J380 

□1 

PI 

•  | 

— 

— 

— 

ill 

II 

Ld 

13  min  17  min  22  min  * 

"hT 

"mUT 

Sec 
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R  •  Regularly 
O  ■  Occasionally 
N  «  Never 


How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest* 


Hr  a  Hours 
Min  a  Minutes 
Sec  a  Seconds 


What  is  Incumbent's 
Performance 
Time? 

(Fill  in  box  below) 


J383 


K389 


K398 


E203 


E206 


E209 


E212 


E213 


F242 


F243 
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R  »  Regularly 

Hr  ■  Hours 

O  ■  Occasionally 

Min  ■  Minutes 

N  ■  Never 

Consensus 

Sec  •  Seconds 

How  Often  You 

Performance  Time  Scale 

What  is  Incumbent's 

Observe  Incumbent 

Pa  Harm  Task  ^ 

Performance 

^  Tirr»? 

rviiVMii  i  nan 

(Check  one  box  below) 

Fastest  Normal  Slowest 

^  i  line  r 

(Fill  in  box  below) 

F257 


F258 


F260 


F261 


H287 


H297 


H311A 


H311B 


H319 


H325 


R 

O 

N 

R 

O 

N 

R 

O 

N 

R 

O 

N 

R 

O 

N 

2  min 
30  MC 


30  mc  1  min  2  min 


15  min  20  min  30  min 


15  min  20  min  30  min 


5  min  7  min  10  min 


3  min 
30  MC 


5  min 


□ 

□ 

CD 

tl 

N 

3  min  5  min  7  min  ^ 

Hr 

Min 

Sec 

LI 

l_| 

1 

tl 

tl 

LlJ 

45  sec  1  min  2  mm 

30  sec 

Hr 

Min 

Sec 

L_| 

LI 

□ 

□ 

N 

^  30  sec  1  min  2  min  * 

Hr 

"mUT 

SeT 

" 

U 

L_| 

M 

tl 

N 

18  3  sec  5  sec  r  sec 

Hr 

"mSIT 

Sec 

R 

o 

N 

**  20  min  30  min  40  min 

Hr 

Min 

Sec 

Hr 

Min 

Sec 

Hr 

Min 

Sec 

Hr 

Min 

Sec 

Hr 

Min 

Sec 

Hr 

Min 

Sec 

62 


R  ■  Regularly 
O  •  Occasionally 
N  «  Never 

How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest* 


Hr  ■  Hours 
Min  m  Minutes 
Sec -Seconds 

What  is  Incumbent's 
Performance 
Time? 

(Fill  in  box  below) 


H326 

H327 

1336 

1339 

1341 

1346 

1347 

□ 

r~| 

r 

□ 

H 

fc 

to  sec  ausec  1  min  '  * 

Hr 

Min 

Sec 

□ 

13 

3 

□ 

13 

E 

^  tusec  tasec  20  sec' 

Hr 

"Min 

Sec 

□ 

□ 

3 

□ 

13 

E 

emin  to  min  ta  mn  ^ 

Hr 

Min 

Sec 

□ 

□ 

3 

3 

13 

E 

4a  sec  t  mm  z  min  ^ 

30  sec  30  sec 

Hr 

Min 

Sec 

□ 

U 

3 

3 

□ 

E 

^  « mm  tamln  25 min' 

~hT 

Min 

Sec 

_| 

3 

3 

3 

3 

t 

^  3  mm  4  min  7  min  ^ 

30  sec 

~hT 

Min 

Sec 

_ 1 

3 

3 

3 

13 

e 

e  4a  sec  t  mm  t  min  ^ 

30  sec 

Hr 

Min 

Sec 

054 

I 

3 

3 

3 

L3 

E 

^  7  mm  t3  mm  zo  min  ”  e 

Hr 

"Min" 

Sec 

1366 

1 

3 

1 

3 

3 

H 

3  mm  a  mm  b  min  "" ** 

Hr 

Min 

Sec 

070 

1 

3 

3 

□ 

N 

^  "  3  min  7  mm  to  min 

30sec  30sec 

~hT 

Min 

Sec 
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R  ■  Regularly 
O  ■  Occasionally 
N  «  Never 


I 

How  Often  You 
Observe  incumbent 
Perform  Task 
(Check  one  box  belov/) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest 


Hr  -  Hours 
Min  ■  Minutes 
Sec b  Seconds 

What  is  Incumbent's 
Performance 
Time? 

(FBI  in  box  below) 


J379 

K390 

M483 

M484 

N492 

LJ 

L_ 

R 

E 

E 

9  mm  izmin  "  ’  15  min  ^ 

Hr 

Min 

Sec 

1— mI 

u 

□ 

E 

Ld 

H 

E 

1  min  3  min  4  min 

30  sec  30  sec 

~hT 

Min 

Sec 

□ 

□ 

□ 

H 

il 

E 

15  min  20  min  30  min  ^ 

Hr 

Min 

Sec 

□ 

□ 

r 

H 

III 

LL 

2  min  3  min  4  min 

Hr 

Min 

Sec 

□ 

□ 

1 

ZJ 

□ 

Ld 

36 sec  1  min  2 min  1  ^ 

Hr 

Min 

Sec 

** 

N493 

□ 

□ 

□ 

ij 

□ 

□ 

30  sec  1  min  -  2  min  •* 

15  sec  30  sec 

~hT 

Min 

Sec 

N499 

□ 

□ 

□ 

il 

□ 

□ 

7  min  12  min  20  min  ^ 

30  sec 

Hr 

Min 

Sec 

In  this  specialty,  consider  the  maximum  amount  of  acceptable  work  that  can  be  done  by  a 
person  in  a  typical  day  as  100  percent.  What  percent  of  the  maximum  could  the  person  you 
are  currently  rating  do  in  a  typical  day?  Write  your  estimate  in  the  box  below. 

Percent  of 
Maximum 

- 1  i - 1 


AEROSPACE  GROUND  EQUIPMENT  (AGE)  SPECIAUST 

AFS  454X1 


Name _ 

Airman's  Name _  Airman's  SSN 


R  »  Regularly 
O  ■  Occasionally 
N  •  Never 

How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest" 


Hr  ■  Hours 
Min  a  Minutes 
Sec  a  Seconds 

What  is  Incumbent's 
Performance 
Time? 

(Fill  m  box  below) 


£120 


F154 


R 

O 

N 

8  min  13  min  21  min ' 


Hr 

Min 

Sec 

FI  55 


FI  62 


G179 


H209 


H215 


H23B 


R 

o 

N 

1 1  mm  1  /  min  *«  min 

Hr 

Min 

Sec 

R 

o 

N 

9  min  — — ~  15  min  23  min 

Hr 

Min 

Sec" 

□ 

p 

P 

H 

M 

H 

15  min  23  min  33  mm  ^ 

Hr 

Mm 

Sec 

□ 

[i] 

n 

td 

M 

M 

^  48  min  1  hr  1  hr 

13  min  39  min 

Hr 

Min 

Sec 

p 

p 

n 

□ 

□ 

'Id 

**  20  min  ""  31  min  4/  min  * 

Hr 

Mm 

Sec 

p 

p 

n 

P 

□ 

td 

E 

*  13  min  19  min  29  min 

E 

Mm 

Sec 

□ 

p 

IP 

1 

□ 

ltd 

It 

**  17  min  25  min  36  min 

E 

Mm 

Sec 
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R  ■  Regularly 
O  -  Occasionally 
N  ■  Never 


How  Often  You 
Observe  incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Hr  *  Hours 
Min  m  Minutes 
Sec  ■  Seconds 


What  is  Incumbent's 
Performance 
Tima? 

(Fill  in  box  below) 


Fastest 


Normal 


Slowest 


R  ■  Regularty 

Hr  ■  Hours 

O  m  Occasionally 

Min  m  Minutes 

N  m  Never 

Consensus 

Sec  >  Seconds 

How  Often  You 

Performance  Time  Scale 

What  is  Incumbent's 

Observe  incumoent 

Performance 

Perform  Task 

■ _  . 

^  Time? 

(Check  one  box  below) 

Fastest  Normal  Slowest- * 

(Fill  in  box  below) 

L421 


M446 


R 

O 

N 

R 

0 

N 

18  mm  35  min  49  min' 


30  min  45  min 


1  hr 
3  mi! 


■ 

■ 

■ 

LL 

Min 

3 

■ 

■ 

■ 

Ll 

Min 

Seel 

N477 


N488 


R 

0 

N 

R 

0 

N 

26  min  —  39  min  54  min ' 


38  min  55  min 


1  hr 
18  min 


■ 

■ 

■ 

Ll 

Min 

3 

■ 

■ 

■ 

Ll 

Min 

Sec  j 

N503 


P549 


P555 


E143 


F153 


F157 


□ 

□ 

r 

■ 

■ 

■ 

H 

□ 

Lu 

9  min  18  min  28  mm 

3 

Min 

Sec 

Ll 

Ll 

| 

■ 

■ 

■ 

tid 

□ 

Ll 

io  min  to  min  23  mm 

3 

Min 

Sec 

u 

L 

■ 

■ 

■ 

tzJ 

tzi 

Li 

28  min  40  min  08  min  ^ 

- 

Min 

Sec 

u 

Ll 

1 

■ 

■ 

■ 

□ 

□ 

Li 

7  mm  12  mm  i6mm  ^ 

3 

Min 

Sec 

Ll 

|_| 

1 

■ 

■ 

■ 

□ 

□ 

Li 

^  10  mm  io  min  ZO  mm 

2. 

Min 

Sec 

Ll 

|_| 

m 

■ 

■ 

□ 

L 

N 

io  min  io  mm  zo  min  ^ 

Hr 

m 

Sec 
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R  •  Regular)" 

O  •  Occasional  y 
N ■  Never 

How  Often  Von 
Observe  Incumbint 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest  Normal  Slowest' 


Hr  a  Hours 
Min  b  Minutes 
Secs  Seconas 

What  is  Incumbent’s 
Performance 
Time? 

(Fill  in  box  below) 


RON 


30  min  45  min 


Hr  I  Min  Sec 


RON 


1  hr 
30  min 


2  hr 
30  min 


Hr  I  Min  I  Sec  I 


10  min ' 


Hr  I  Min  I  Sec  j 


o 

N 

10  min  15  min  25  min' 


Hr  I  Min  |  Sec  I 


O 

N 

15 min  30 min' 


10  min  '  20  min' 


Hr  I  Min  I  Sec 


Hr  Min  Sec 


RON 


10  min  11  15  min' 


Hr  I  Min  I  Sec 


RON 


12  min  *“ ““ "  17  min 


Hr  Min  Sec 


RON 


30  min  45  min 


Hr  I  Min  I  Sec 


RON 


1  hr 
30  min 


2  hr  ' 
30  min 


Hr  Min  Sec 
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J347 


J355 


L406 


L436 


1437 


M444 


R  ■  Regularly 
O  -  Occasionally 
N •  Never 

How  Often  You 
Observe  Incumben 
Perform  Task 
(Check  one  box  belo  v) 


Consensus 

Performance  Time  Scale 


Fastest  Normal  Slowest 


Hr  ■  Hours 
Min  •  Minutes 
Sec  «  Seconds 

What  is  Incumbent's 
Performance 
Time? 

(FIJI  in  box  below) 


□ 

□ 

□ 

d 

d 

d 

30  min  45  min  l  nr 

20  min 

Hr 

Min 

Sec 

□ 

□ 

□ 

□ 

td 

d 

^  25  min  40  min  1  hr 

”hT 

Min 

Sec 

□ 

□ 

r 

d 

td 

d 

'  12  mm  30  min  *5  min 

Hr 

Mm 

Sec 

□ 

□ 

r 

□ 

id 

d 

m  15  min  25  min  35  min  ^ 

Hr 

Mm 

Sec 

□ 

□ 

r 

td 

d 

d 

"*  5  min  "  10  min  15  min 

Hr 

Mm 

See 

R 

O 

N 

30  min  1  hr 


1  hr 
30  min 


Hr 

Mm 

Sec 

M447 


N475 


N486 


N487 


□ 

□ 

□ 

_J 

d 

d 

d 

—  10  min  20  mm  3u  min  “** 

Hr 

Mm 

lecj 

□ 

□ 

□ 

1 

d 

d 

d 

^  15  min  ™"“—  20  min  30  min 

Hr 

i  Mml 

Sec 

□ 

□ 

□ 

” Hr 

Mm 

Sec 

d 

d 

d 

I  nr  §  nr  1  ni 

20  min  45  min 

' 

□ 

□ 

ll_ 

d 

Id 

Id 

—  20  min  30  min  45  min 

Hr 

Mm 

Sec 
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N494 


P554 


R  ■  Regularly 
O  b  Occasionally 
N- Never 

How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest 


Hr  e  Hours 
Min  a  Minutes 
Sec  a  Seconds 


□ 

□ 

□ 

□ 

□ 

□ 

^  7  min  15  min  20  min  ^ 

Hr 

Min 

Sec 

□ 

□ 

□ 

□ 

□ 

□ 

*  16  min  17  min  20  min 

Hr 

Min 

Sec 

In  this  specialty,  consider  the  maximum  amount  of  acceptable  work  that  can  be  done  by  a 
person  in  a  typical  day  as  1 00  percent.  What  percent  of  the  maximum  could  the  person  you 
are  currently  rating  do  in  a  typical  day?  Write  your  estimate  in  the  box  below. 


Percent  of 
Maximum 


1 _ _ _ 1 

1 

1% 

1 

100% 

% 
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AVIONIC  COMMUNICATION  AND  NAVIGATION  SYSTEMS 

SPECIALIST 

AFS  455X2 


Name 


Airman's  Name 


Airman’s  SSN 


R  -  Regularly 
O  •  Occasionally 

N- Never 

How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest* 


Hr  •  Hours 
Min  *  Minutes 
Sec*  Seconds 

What  is  Incumbent’s 
Performance 
Time? 

(Fill  in  box  below) 


E110 


E138 


E140 


F169 


F196 


FI  99 


F200 


F206 


"*!1°  iu  mm  15  mm  20  min  "" 

Hr 

Min 

Sec 

2  mm  4  min  6mn  ^ 

Hr 

Min 

Sec 

“ 

m  8  mm  to  mm  12  min  ^ 

m5T 

Sec 

m  15  mm  zomin  aomm 

z 

"MirT 

Sec 

J 

1,1  izmin  1 1  min  22  min 

z 

Min 

Sec 

3  min  s  mm  i  mm  ^ 

Hr 

Min 

Sec 

3  mm  5  mm  7  min 

~hT 

Min 

Sec 

**'  5  min  "  10  min  is  min 

"hT 

Min 

Sec 
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R  ■  Regularly 

Hr  ■  Hours 

O  ■  Occasionally 

Min  a  Minutes 

N ■  Never 

Consensus 

Sec  a  Seconds 

How  Often  You 

Performance  Time  Scale 

What  is  Incumbent's 

Observe  Incumbent 

Partarm  Task  ^ 

Performance 

^  Time? 

w  viiwiMi  man 

(Check  one  box  below) 

s"*“'  Fastest  Normal  Slowest”””® 

►  1  HI  ID  f 

(FUI  in  box  below) 

G218 


G227C 


G232 


G250 


G252 


G262 


L398 


El  09 


E139C 


E139N 


□ 

P 

p 

H 

P 

fcj 

^  30  mm  t  hr  1  hr 

30  min 

Hr 

Min 

Sec 

LI 

P 

P 

□ 

□ 

□ 

40  mm  t  hr  1  1  1  hr  1  “  ^ 

30  min 

Hr 

Min 

Sec 

U 

P 

p 

□ 

Q 

E 

aumm  oamin  ’  ™  '  "  ^ 

10  min 

Hr 

Min 

Sec 

p 

P 

P 

□ 

□ 

E 

4  8  min  iz  min  ’  15  min  m 

Hr 

Min 

Sec 

p 

P 

| 

□ 

El 

ill 

6  min  .  10  min  15  min 

~hT 

Min 

Sec 

p 

P 

|  i 

tj 

□ 

E 

^  a  min  e  mm  b  min 

Hr 

Min 

Sec 

p 

P 

l_ 

E 

EJ 

N 

7  min  id  mm  is  min 

~hT 

Min 

Sec 

P 

P 

p 

m 

□ 

E 

1  min  2  min  3  min 

Hr 

Min 

Sec 

P 

L 

M 

E 

E 

—  2  min  3  min  6  min  * 

”hT 

Min 

Sec 

R 

0 

N 

1  mm  2  min  3  min 

Hr 

Min 

Sec 
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F164 


FI  67 


FI  85 


FI  87 


F202 


F209 


G208 


G215 


G222 


G226 


R  ■  Regularly 
O  •  Occasionally 
N  •  Never 

How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


R 

O 

N 

R 

O 

N 

R 

O 

N 

R 

O 

N 

R 

O 

N 

O 

N 

Consensus 

Performance  Time  Scale 


Fastest  Normal  Slowest' 


2  min  — —  4  min 


30  min  1  hr 


7  min 


10  min  15  min  — —  20  min 


5  min  10  min  12  min' 


4  min  8  min  12  mir 


5  min  10  min  12  min 


1  hr 
30  min 


Hr  m  Hours 
Min  ■  Minutes 
Sec  •  Seconds 

What  is  Incumbent's 
Performance 
Time? 

(Fill  in  box  below) 


□ 

□ 

□ 

□ 

tl 

□ 

H 

’'**  5  mm  to  min  15  min  ^ 

Hr 

Min 

Sec 

1— -f 

|_| 

n 

| _ 1 

tl 

If] 

M 

^  20  mm  30  min  40  min  ^ 

~hT 

Min 

Sec 

II 

□ 

□ 

tl 

tl 

H 

^  6  min  a  mm  12  rnn ' 

Hr 

Min 

SeT 

- tl— 

- |J — | 

LI 

U 

u 

□ 

tl 

H 

"te  1  nr  2  nr  3  hr  ^ 

30  min 

Hr 

Min 

Sec 

Hr 

Min 

Sec 

Hr 

Min 

Sec 

Hr 

Min 

Sec 

Hr 

Min 

Sec 

Hr 

Min 

Sec 

Hr 

Min 

Sec 
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R  m  Regularly 
O  «  Occasionally 
N •  Never 

How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest  Normal 


Slowest 


Hr  m  Hours 
Min  ■  Minutes 
Sec  •  Seconds 

What  is  Incumbent's 
Performance 
Time? 

(Fill  in  box  below) 


G227N 


G229 


G236 


G239 


G256 


G258 


G259 


1302 


L404 


L407 


□ 

□ 

n 

- 

td 

□ 

Ld 

^  5  mm  to  mm  20  mm  ^ 

"hT 

Min 

Sec 

M 

1  | 

n 

M 

M 

1  min  3  min  5  min 

Hr 

Min 

SeT 

□ 

□ 

□ 

FI 

M 

M 

a  min  to  min  1b  min  ^ 

Hr 

Min 

Sec 

|_J 

□ 

□ 

H 

0 

23  min  30  mm  40  mm  ^ 

Hr 

Min 

Sec 

□ 

1 

n 

o 

N 

2  min  4  min  6  min  ~  *'“* 

Hr 

"Min 

Seel 

□ 

r 

H 

td 

r 

—  7  min  10  min  15  min  ^ 

Hr 

Min 

Sec 

□ 

□ 

r 

H 

M 

r 

5  min  e  min  lb  min  **“ 

Hr 

Min 

Sec 

□ 

□ 

n 

td 

td 

□ 

5  min  to  min  1b  min 

Hr 

Min 

Sec 

□ 

□ 

n 

L 

td 

td 

t 

^  4  mm  e  min  e  min 

t 

Min 

Sec 

□ 

r 

t 

o 

t 

3  min  5  min  tOmin  ^ 

Hr 

Min" 

Sec 
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R  -  Regularly 

Hr  b  Hours 

O  ■  Occasionally 

Min  b  Minutes 

N  ■  Never 

Consensus 

Sec  b  Seconds 

How  Often  You 

Performance  Time  Scale 

What  is  Incumbent's 

Observe  Incumbent 

Performance 

Perform  Task  ^ 

• 

►  Time? 

(Fill  in  box  below) 

r  oil wiiii  i  aen 

(Check  one  box  belov/) 

Fastest  Normal  Slowest 

M461 


M483 


U971 


3  min 


6  min 


10  min 


8  min 


10  min 


20  min 


9  min 


12  min 


18  min ' 


In  this  specialty,  consider  the  maximum  amount  of  acceptable  work  that  can  be  done  by  a 
person  in  a  typical  day  as  1 00  percent.  What  percent  of  the  maximum  could  the  person  you 
are  currently  rating  do  in  a  typical  day?  Write  your  estimate  in  the  box  below. 


1%  100% 


Percent  of 
Maximum 


% 
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PERSONNEL  SPECIALIST 

AFS  732X0 


Name 


Airman's  Name 


Airman's  SSN 


R  •  Regularly 
O  ■  Occasionally 
N  b  Never 

How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest 


Hr  b  Hours 
Min  b  Minutes 
Sec  b  Seconds 

What  is  Incumbent's 
Performance 
Time? 

(Fill  in  box  below) 


E35 


E102 


E107 


El  14 


F140 


L733 


0826 


R1011 


□ 

R 

O 

N 

7  min  10  min  12  min  ^ 

Hr 

Min 

Seel 

1 

R 

O 

N 

3u  sec  1  min  i  nun 

30  sec 

Hr 

Min 

Cee 

wev 

R 

O 

N 

3  sec  15  tec  30  sec 


Hr 

Min 

Sec 

R 

O 

N 

1  min  2  min  3  min 


Hr 

Min 

Sec 

R~ 

O 

N 

45  sec 


1  min 
30  tec 


2  min 


Hr 

Min 

Sec 

R 

O 

N 

2  min  3  min  5  min 


Hr 

Min 

Sec 

R 

O 

N 

7  min  10  min  15  min' 


Hr 

Mm 

Sec 

R 

O 

N 

45  tec  1  min 


2  min 


Hr 

"Mm' 

Sec 
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R  ■  Regularly 
O  b  Occasionally 
N  b  Never 


Hr  b  Hours 
Min  b  Minutes 
Sec  b  Seconds 


How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest' 


What  is  Incumbent's 
Performance 
Time? 

(Fill  In  box  below) 


A4 


A10 


A19 


B30 


B37 


B38 


B41 


B44 


E95 


E97 


□ 

p 

p 

□ 

Ml 

y 

2  mm  4  mm  t>  min  ^ 

Hr 

Min 

Sec 

□ 

P 

p 

y 

M 

y 

^  3  min  5  min  10  min 

Hr 

Min 

Sec 

p 

P 

p 

H 

M 

y 

1  nr  1  hr  1 "  "  -nr  m 

30  min 

"hT 

Min 

Sec 

M 

P 

p 

n 

H 

y 

98  5  mm  9  mm  id  mm  ^ 

"hT 

Min 

Sec 

p 

p 

M 

M 

y 

5  mm  '  io  mm  zomin  ^ 

~ 

"Min" 

Sec 

p 

p 

p 

H 

H 

y 

do  nr  102  hr  i/6hr 

Hr 

Min 

Sec 

P 

□ 

M 

m 

y 

45  min  i  nr  t  nr  ^ 

30  min 

Hr 

Min 

Sec 

p 

P 

p 

y 

□ 

IM 

5  min  io  mm  tomin  980 

Hr 

Min 

Sec 

□ 

p 

P 

j 

□ 

□ 

IM 

3  mm  5  mm  v  min 

’"Hr  ' 

Min 

Sec 

□ 

P 

1  r 

y 

y 

E 

10  mm  15  mm  zomin”"""15*'" 

Hr 

Min 

Sec 

77 


R  -  Regularly 
O  ■  Occasionally 
N  •  Never 

How  Often  You 
Observe  Incumbent 
Perform  Task 
(Check  one  box  below) 


Consensus 

Performance  Time  Scale 


Fastest 


Normal 


Slowest 


Hr  ■  Hours 
Min  *  Minutes 
Sec  •  Seconds 

What  is  Incumbent's 
Performance 
Time? 

(Fill  In  box  below) 


0850 


Z1452 


Z1466 


Z1494A 


Z1494B 


Z1495 


R 

O 

N 

□ 

□ 

n 

r 

td 

Id 

c  i5min  IB  mm  20  min 

t 

Min 

Sec 

□ 

p 

1 

| 

H 

H 

Id 

^  3  min  4  min  5  min  * 

Hr 

Min 

Sec 

□ 

p 

td 

td 

M 

***"  2  min  3  min  — —  4  niin  m 

Hr 

Min 

SeT 

' 

□ 

p 

□ 

til 

n' 

5  min  /min  iSmin 

Hr 

Min 

Sec 

□ 

p 

n 

□ 

H 

N 

5  min  — —  7  min  — —  15  min  ^ 

Hr 

Min 

Sec 

2  min  3  min  5  min 


Hr 

Min 

Sec 

Z1 51 1 


Z1533 


R 

O 

N 

R 

O 

N 

2  min  4  min  7  min 


15  min  20  min  30  min' 


Hr 

Min 

Sec 

Hr 

Min 

Sec 

In  this  specialty,  consider  the  maximum  amount  of  acceptable  work  that  can  be  done  by  a 
person  in  a  typical  day  as  100  percent.  What  percent  of  the  maximum  could  the  person  you 
are  currently  rating  do  in  a  typical  day?  Write  your  estimate  in  the  box  below. 


1%  100% 


Percent  of 
Maximum 


% 
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APPENDIX  E 


Generalized  Motivation  Scale  (GMS) 


81 


GENERAL  SCALE 


Instructions 


To  familiarize  yourself  with  this  questionnaire,  it  would  be  useful  to  scan  it  quickly 
before  answering  any  of  the  items.  It  is  a  good  idea  to  work  rather  fast  and  to  give  your 
first  reaction  to  each  item.  Some  of  the  statements  may  seem  alike  to  you,  but  all  of 
them  are  necessary  to  show  slight  differences  of  opinion.  Because  your  opinion  is  being 
asked  for,  there  are  no  right  or  wrong  answers.  Also,  keep  in  mind  that  the  responses  to 
this  questionnaire  are  being  kept  confidential,  so  answer  items  as  honestly  as  you  can. 

Please  do  not  make  any  marks  in  this  booklet.  You  should  record  your  answers 
on  Pan  2  -  General  Scale,  located  on  Side  2  of  the  separate  answer  sheet  Mark  the 
circle  on  your  answer  sheet  that  corresponds  with  your  opinion  using  the  following  scale: 

Blacken  the  circle  marked  "A"  if  you  strongly  disagree  with  the  statement. 

Blacken  the  circle  marked  "B"  if  you  disagree  with  the  statement 

Blacken  the  circle  marked  "C"  if  you  slightly  disagree  with  the  statement 

Blacken  the  circle  marked  "D"  if  you  neither  agree  nor  disagree  with  the  statement 

Blacken  the  circle  marked  "E"  if  you  slightly  agree  with  the  statement 

Blacken  the  circle  marked  "F"  if  you  agree  with  the  statement 

Blacken  the  circle  marked  "G"  if  you  strongly  agree  with  the  statement 

The  scale  is  printed  at  the  top  of  the  following  two  pages  for  reference.  When  you 
are  told  to  begin,  read  through  the  questionnaire  items  one  at  a  time  and  begin  recording 
your  answers  on  the  "General  Scale"  section  on  the  answer  sheet 
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Please  use  the  following  scale  when  responding  to  each  item: 


A  B  C  D  E  F  G 

Strongly  Disagree  Slightly  Neither  Slightly  Agree  Strongly 

Disagree  Disagree  Agree  Agree 


1.  When  faced  with  a  difficult  problem,  1  am  a  less  persistent  worker  than  my  peers. 

2.  If  I  fail  at  something,  it’s  because  I  did  not  make  plans  in  advance. 

3.  Others  tend  to  evaluate  me  as  being  competent  in  most  areas. 

4.  Someone  usually  has  to  tell  me  what  to  do. 

5.  Setting  goals  is  not  important  for  success. 

6.  Failure  makes  me  quit  what  I  am  doing  to  try  something  entirely  different. 

7.  When  someone  tells  me  that  I  did  a  good  job,  it’s  because  I  did  do  a  good  job. 

8.  When  I  work  on  a  long  task,  I  have  difficulty  finishing  it  successfully. 

9.  It  is  important  to  set  goals  in  order  to  be  successful 

10.  If  something  looks  too  complicated,  I  will  not  bother  to  try  it 

11.  When  I  evaluate  my  performance  in  general  I  find  it  to  be  superior. 

12.  I  like  working  on  problems  that  I  know  I  can  solve  better  than  trying  problems  that 
are  new  to  me. 

13.  It  is  not  necessary  to  make  plans  in  order  to  succeed. 

14.  I  rarely  achieve  goals  that  I  have  set  for  myself. 

15.  When  I  make  plans  to  solve  problems,  I  am  certain  to  make  them  work. 

16.  I  can  work  at  something  for  a  very  long  time  if  necessary  in  order  to  finish  it 

17.  I  have  to  set  goals  before  starting  a  project 

18.  On  the  whole,  I  am  more  competent  than  my  peers. 


83 


Please  use  the  following  scale  when  responding  to  each  item: 


A  B  C  D  E  F  G 

Strongly  Disagree  Slightly  Neither  Slightly  Agree  Strongly 

Disagree  Disagree  Agree  Agree 


19.  Failure  does  not  spur  me  on  to  try  new  and  better  solutions  to  a  problem. 

20.  I  feel  that  I  need  to  be  capable  at  my  job  in  order  to  be  successful. 

21.  Others  tend  to  evaluate  me  as  being  very  competent. 

22.  Most  of  the  time,  I  do  not  expect  to  be  successful. 

23.  I  believe  that  it  is  important  to  finish  a  job,  even  if  it  is  not  challenging  to  me. 

24.  I  expect  to  be  successful  most  of  the  time. 

25.  I  admire  people  who  try  to  learn  something  new  everyday. 

26.  I  usually  do  not  feel  very  competent  when  attempting  very  difficult  problems. 

27.  Setting  goals  is  not  important  to  me. 

28.  I  am  usually  successful  when  I  strive  to  accomplish  a  goal. 

29.  I  believe  that  learning  new  things  can  help  me  perform  better. 

30.  I  cannot  work  at  something  difficult  for  a  very  long  time  even  if  it  is  necessary  in 

order  to  finish  it.  c 
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APPENDIX  F 


Task  Timing  Forms 
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TIMING  FORM  -  AIRCREW  LIFE  SUPPORT  (AFS  122X0) 


Examinee  Name 


Grade 


Middle  Initial 


Location  (AFB). 


Task  E199  Make  entries  on  AFTO  Form  152  (chemical  ensemble  inspection  record 

R 

Time: _ . _ . _  Cut  off:  8  mins.  24  secs.  Rating:  12  3  4 


Task  H295  Fit  the  55/P  helmet  using  a  custom  liner 


Time: _ . _ . 


Cut  off:  12  minutes 


Rating:  12  3  4 


Task  H303  Perform  a  30  day  inspection  on  an  HGU-55/P  helmet 


Time: _ . _ . 


Cut  off:  20  mins.  24  secs. 


Task  H315  Replace  the  nape  strap  and  pad 


Time: 


Cut  off:  4  mins.  48  secs. 


Task  H320  Remove  and  replace  headsets  in  helmet 


Time: _ . 


Cut  off:  3  mins.  36  secs. 


Task  D30  Size  and  fit  oxygen  masks 


Time: 


Cut  off:  6  minutes 


Rating:  12  3  4 


Rating:  12  3  4 


Rating:  12  3  4 


Rating:  1  2  3  4  5 


Task  1349  Perform  oxygen  mask  periodic  inspections 


Time: _ . _ 


Cut  off:  18  minutes 


Rating:  12  3  4 


Task  J380  Remove  and  Install  the  filter  elements  in  the  CRU-80/P 


Time: 


Cut  off:  26  mins.  24  secs. 


Rating:  12  3  4 


Task  J383  Perform  the  mask  exchange  in  the  vapor  hazard  area 


Time: _ . _ . 


Cut  off:  6  mins.  36  secs. 


Task  K389  Fit  and  adjust  a  parachute  harness 


Time: _ . _ . 


Cut  off:  4  mins.  48  secs. 


Task  K398  Perform  30  day  routine  parachute  inspection 


Tune: - - - 


Cut  off:  30  minutes 


Rating:  12  3  4 


Rating:  12  3  4 


Rating:  12  3  4 


O  N 


O  N 


O  N 


O  N 


O  N 


O  N 


O  N 


O  N 


O  N 


O  N 


O  N 
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TIMING  FORM  -  AEROSPACE  GROUND  EQUIPMENT  (AFS  4S4X1) 


Examinee  Name 

Grade _ 


SSN 


Last  First 

AFS  _ 


Middle  Initial 


Location  (AFB). 


Task  F154  Perform  an  aircraft  support  generator  service  inspection 


RON 


Time: 


Cut  off:  29  mins.  2  secs. 


Rating:  1  2  3  4  5 


Task  F155  Perform  a  service  inspection  on  a  load  bank 


RON 


Time: 


Cut  off:  27  minutes 


Rating:  1  2  3  4  5 


Task  F162  Perform  a  service  inspection  on  a  hydraulic  test  stand 


RON 


Time: 


Cut  off:  40  minutes 


Rating:  1  2  3  4  5 


Task  1284  Remove  and  replace  an  alternator  belt 


RON 


Time: _ . _ . 


Cut  off:  41  mins.  2  sec. 


Rating:  1  2  3  4  5 


Task  1300  Replace  the  Rare  fitting  on  a  fuel  line 


RON 


Time: 


Cut  off:  39  minutes 


Rating:  1  2  3  4  5 


Task  P549  Perform  an  operator's  inspection  of  an  AF  vehicle,  completing  AFTO  Form  373 

RON 

Tune: _ . _ . _  Cut  off:  27  minutes  Rating:  1  2  3  4  5 
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TIMING  FORM  -  AVIONIC  COMMUNICATION  AND  NAVIGATION  (AFS  455X2) 


Examinee  Name  _ 

SSN 

Grade 

Last  First 

AFS 

Middle  Initial 

Location  (AFB). 

Task  E110  (173)  Research  and  identify  information  regarding  parts  using  technical  data 

RON 

Time: _ . _ . _  Cut  off:  24  minutes  Rating:  1  2  3  4  5 


Task  E138  (160)  Complete  AFTO  Form,  Maintenance  Data  Collection  Record 

RON 

Time: _ . _ . _  Cut  off:  7  mins.  12  secs.  Rating:  1  2  3  4  5 


Task  F169  (218)  Visually  inspect  receiver-transmitter  (RT)  units  or  other  radio/test 
equipment  received  from  supply  or  manufacturers 

RON 

Time: _ . _ . _  Cut  off:  36  minutes  Rating:  1  2  3  4  5 


Task  F199  (233)  Safety  wire  system  components 
Time: _ . _ . _  Cut  off:  8  min.  24  sec. 


RON 

Rating:  1  2  3  4  5 


Task  F200  (234)  Properly  set  up  a  flightline  maintenance  stand 
Time: _ . _ . _  Cut  off:  8  mins.  24  secs.  Rating:  1 


RON 
2  3  4  5 


Task  F206  (238)  Trace  circuits  or  signals  using  aircraft  wiring  diagrams 

RON 

Time: _ . _ . _  Cut  off:  18  minutes  Rating:  1  2  3  4  5 


Task  G262  (260) 

Time: _ . _ . 

Task  G218  (250) 

Time: _ . _ . 

Task  L398  (539) 
Time: - - - 


Set  up  ultra  high  frequency  test  equipment 


RON 


Cut  off:  9  mins.  24  secs.  Rating:  1  2  3  4  5 


Perform  a  minimum  standards  (bench)  check  on  an  ultra  high  frequency 
(UHF)  receiver-transmitter  (RT) 

RON 

_  Cut  off:  1  hour  48  mins.  Rating:  1  2  3  4  5 


Isolate  malfunctions  in  interphone  cords 


Cut  off:  18  minutes 


RON 

Rating:  1  2  3  4  5 
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TIMING  FORM  -  PERSONNEL  (AFS  723X0) 


Examinee  Name 

SSN 

Grade 

Last  First 

AFS 

Middle  Initial 

Location  (AFBi 

Task  E35 

Draft  a  message  saying  that  a  job  inventory  was  mailed 

R 

0 

N 

Time: _ . _ .  _ 

_  Cut  off:  14  mins.  24  secs. 

Rating:  12  3  4 

5 

Task  E102  (116) 

Construct  immediate  inquiries  to  determine  date  of  rank,  unit  (PAS),  and 

duty  phone  for  someone  stationed  at  your  base 

R 

0 

N 

Time:  .  . 

_  Cut  off:  1  mins.  48  secs. 

Rating:  12  3  4 

5 

Task  E107  (121) 

Dispose  of  unclassified  PDS  products  of  only  a  few  pages 

R 

0 

N 

Time: _ . _ .  _ 

_  Cut  off:  36  seconds 

Rating:  12  3  4 

5 

Task  E114  (131) 

Sign  in  a  document  and  sign  out  again,  using  AF  Form  614 

R 

0 

N 

Time: _ . _ .  _ 

_  Cut  off:  3  min.  36  sec. 

Rating:  12  3  4 

5 

Task  F140 

Open  and  close  a  CRT  in  the  CBPO 

R 

0 

N 

Time: _ . _ .  _ 

_  Cut  off:  2  mins.  24  secs. 

Rating:  12  3  4 

5 

Task  L733 

Rev  ond  to  a  request  to  release  information  from  a  member’s  file  in 

accordance  with  the  Privacy  Act 

R 

0 

N 

Time: _ . _ .  _ 

_  Cut  off:  6  minutes 

Rating:  1  2  3  4  5 

Task  0826  (719) 

Property  file  10  documents  in  personnel  records 

R 

0 

N 

Time: _ . _ .  _ 

_  Cut  off:  18  minutes 

Rating:  12  3  4 

5 

Task  R1011  (896)  Compute  service  dates 
Time: _ . _ _ Cut  off:  2  mins.  24  secs. 


RON 

Rating:  1  2  3  4  5 


APPENDIX  G 


Supervisor  Time  Estimation  Session  Instructions 
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SUPERVISOR  TIME  ESTIMATION  WORKSHOPS 


Introduction 


Good  morning,  my  name  is _ and  I’m  with  the  Human  Resources  Research 

Organization.  We  are  a  contractor  hired  by  die  Air  Force  to  examine  better  ways  of  selecting  new  airmen 
and  placing  diem  in  appropriate  jobs.  Since  we  are  not  experts  in  the  work  you  do,  your  participation  in 
this  study  is  extremely  important  in  helping  us  improve  how  the  Air  Force  manages  its  people.  We  have 
selected  four  Air  Force  specialties  for  data  collection  and  analysis:  4  22X0  (Aircrew  Life  Support);  454X1 
(AGE);  455X2  (Avionic  Communication  and  Navigation);  and  732X0  (Personnel).  Everyone  here  should 
be  in  one  of  these  four  specialties. 

Basically,  what  we  will  be  asking  you  to  do  this  morning  is  to  estimate  how  long  it  would  take 
some  of  the  airmen  who  work  for  you  to  accomplish  some  of  die  tasks  in  your  specialty.  We  realize  that 
this  is  not  an  easy  thing  to  do,  but  we  will  give  you  some  instructions  and  materials  that  will  hopefully 
make  it  easier.  There  are  no  right  or  wrong  answers  here,  we  simply  want  your  best  judgments  as  experts 
working  in  your  various  fields.  You  have  the  most  experience  working  with  your  people  and  know  best 
what  their  capabilities  and  limitations  are.  While  you  are  here  this  morning  your  airmen  are  in  another 
room  taking  a  series  of  written  tests.  Later  on  this  week  we  will  be  visiting  some  of  your  duty  sections 
to  observe  some  of  your  people  at  work. 

Before  we  go  on  are  there  any  questions? 

Background  Information  Forms,  Privacy  Act  Statements  and  Supervisor-Airmen  Forms 

We’d  like  to  begin  by  handing  out  a  batch  of  forms  to  each  supervisor  here  today.  I  will  read 
down  through  the  list  of  supervisors.  Please  raise  your  hand  when  your  name  is  called.  [Hand  them  a 
batch  of  forms  including  Supervisor-Airmen  List,  Background  Information  Form,  Tasks  for  Time 
Estimation  Booklets,  and  one  Time  Estimation  Form.  Note  attendance  on  the  master  list  of 
supervisors.]  Is  there  anybody  whose  name  I  did  not  call?  [If  supervisor  is  not  on  list,  find  out  if 
he/she  is  replacing  one  of  the  supervisors  on  the  list  Ask  the  supervisor  whose  name  is  not  on  the 
list  if  his/her  airmen  are  in  the  airmen  testing  session.  Give  him  a  blank  Supervisor-Airmen  List 
and  help  him/her  fill  the  form  out]  Did  anyone  not  get  a  batch  of  forms?  Also,  we  have  a  supply  of 
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pencils  if  anyone  needs  one.  All  the  fonns  you  will  fill  out  today  may  be  done  in  either  pencil  or  pen. 


The  first  sheet  you  received  was  the  Supervisor- Airmen  List  that  names  die  airmen  for  whom  you 
will  be  making  your  time  estimates.  We  need  to  be  sure  that  we  have  the  right  information  concerning 
the  airmen  who  work  for  you.  This  sheet  should  identify  you  at  the  top  of  the  form  and  then  list  some 
of  die  airmen  who  work  for  you.  Please  be  sure  that  the  top  information  is  correct  Mark  any  corrections 
necessary,  and  then  review  the  information  at  die  bottom.  Also,  in  the  far  right-hand  column  we  need  to 
know  how  many  months  you  have  supervised  each  airman.  Please  fill  in  the  right  hand  column  at  this 
time.  [Pause.]  Does  anyone  see  someone  on  their  form  who  does  not  work  for  them?  [If  so,  call  out 
the  name  to  the  supervisors  from  that  particular  specialty  to  see  if  one  of  them  supervises  the 
airman  in  question.  If  one  of  the  supervisors  does  supervise  the  airman  in  question,  then  have  the 
supervisor  who  supervises  him/her  add  the  airman's  name  to  the  Supervisor-Airman  List.  If  more 
than  one  supervisor  supervises  the  airman  in  question,  then  add  the  airman’s  name  to  the  first-line 
supervisor  with  the  fewest  airmen  to  rate.  Then  have  the  supervisor  who  does  not  supervise  the 
airman  draw  a  line  through  the  airman’s  name  on  the  Supervisor-Airman  List.] 

[Hand  out  task  booklets  with  forms.] 

The  next  sheet  should  be  a  Background  Information  Form  with  a  Privacy  Act  Statement  on  it 
Please  read  the  Privacy  Act  Statement  and  take  a  few  minutes  to  fill  out  die  form.  [Wait  until  all  appear 
to  be  finished. 

Task  for  Time  Estimation  Booklets  and  Time  Estimation  Forms 

Next  you  will  see  a  Task  for  Time  Estimation  Booklet  for  your  specialty.  Check  the  front  cover 
of  die  booklet  Does  everyone  have  the  correct  booklet  for  their  specialty?  Please  do  not  marie  in  these 
booklets,  we  need  to  use  them  again  at  other  bases  we  visit  You  should  also  have  a  Time  Estimation 
Form  for  the  first  airman  listed  on  the  Supervisor-Airmen  List 

First  I’d  like  to  talk  a  little  about  the  Task  for  Tune  Estimation  Booklets.  We  have  tried  to 
identify  a  set  of  tasks  in  each  of  your  specialties  that  are  commonly  performed  by  airmen  and  junior 
NCOs.  dearly,  these  are  not  all  of  the  tasks  you  do,  only  a  representative  sample.  Each  task  is 
numbered,  followed  by  a  description  of  the  task.  You  will  notice  that  the  tasks  are  described  in  quite 
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different  levels  of  detail.  The  first  group  of  tasks  in  your  booklets  have  very  specific  descriptions  and 
steps  specified  and  even  include  the  instructions  you  might  give  someone  if  you  wanted  to  actually  watch 
them  doing  the  task.  The  second  set  of  tasks,  which  appear  towards  the  end  of  your  booklets,  describe 
the  tasks  in  much  less  detail  One  of  the  questions  we  are  trying  to  answer  is  how  much  task  detail  is 
necessary  to  estimate  performance  time  accurately.  All  we  ask  is  that  you  take  your  best  shot  at 
estimating  your  airman’s  performance  time  on  each  task  with  whatever  level  of  detail  is  provided.  Do 
you  have  any  questions  about  the  Task  for  Time  Estimation  Booklets? 

Now  let’s  turn  to  the  Time  Estimation  Forms.  The  top  of  the  form  includes  a  place  for  you  to 
record  your  name,  the  person’s  name  you  are  rating,  and  their  Social  Security  number-you  will  find  these 
on  the  Supervisor-Airmen  Lists  you  reviewed  earlier.  Please  fill  out  the  top  of  the  form  at  this  time. 
[PAUSE.] 

Please  open  your  Task  Booklet  to  the  first  page  and  look  at  the  instructions  for  filling  out  the 
Time  Estimation  Form.  Read  along  silently  as  I  read  aloud. 

"The  first  column  of  the  Time  Estimation  Form  is  a  list  of  task  numbers.  These  numbers 
correspond  to  the  task  numbers  in  your  Task  Booklets.  The  task  numbers  on  the  Time  Estimation 
Form  and  the  Task  Booklets  are  in  the  same  order.  The  next  set  of  columns  asks  you  to  tell  us 
how  often  you  have  observed  your  airmen  doing  each  of  the  tasks.  You  provide  this  information 
by  checking  die  box  above  the  "R,"  "O,"  or  "N"  for  each  task.  "R"  means  you  regularly  see  him 
(or  her)  do  the  task,  "0"  means  you  have  seen  him  do  (he  task  occasionally,  and  "N"  means  you 
have  never  seen  him  do  the  task. 

The  next  section  presents  a  time  line  with  three  time  estimates.  "Fastest"  refers  to  the  fastest  this 
task  could  be  done  and  still  meet  an  acceptable  level  of  performance.  "Normal"  refers  to  the 
ordinary  time  the  average  airman  needs  to  do  this  task  to  the  accepted  level  of  performance. 
"Slowest"  refers  to  the  longest  amount  of  time  that  could  be  allowed  for  performance  of  this  task 
without  negative  consequences  to  foe  job.  These  time  estimates  were  provided  by  six  NCOs  from 
your  career  field.  These  are  provided  for  your  information  only  and  should  not  limit  the  estimates 
you  make  for  your  airmen.  If  you  think  one  of  your  airmen  could  do  a  task  foster  than  the 
"fastest"  time  provided,  please  put  down  what  you  think.  If  you  think  one  of  your  airmen  would 
take  longer  than  foe  "slowest"  time,  put  that  down  as  well.  Or  if  you  estimate  that  your  airman 
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may  perform  more  quickly  or  slowly  than  the  "normal”  time,  record  that  time.  You  are  free  to 
record  any  time  you  think  appropriate  for  your  airmen  on  each  task. 

The  last  column  on  this  form  is  where  you  record  your  estimate  of  how  long  each  of  your  airmen 
would  take  to  do  each  of  the  tasks.  Record  your  time  in  hours,  minutes,  and  seconds,  writing  in 
the  appropriate  number  in  the  boxes  above  "Hr,"  "Min,"  and  "Sec."  If  you  have  never  seen  one 
of  your  airman  perform  a  task,  please  give  your  best  estimate  of  how  long  it  would  take  him  or 
her.  Base  your  estimate  on  your  knowledge  of  his  or  her  performance  on  similar  tasks  in  your 
specialty. 

As  an  example,  think  about  die  task  of  "Starting  a  Car."  Assume  that  you  are  estimating  how 
long  it  would  take  an  airman  to  start  a  motor  pool  car  and  that  die  airman  has  to  check  all  minors, 
gauges,  and  seat/steering  wheel  positions.  Look  at  the  completed  time  estimation  example  below. 
The  box  above  "R"  was  checked  since  you  have  regularly  observed  the  person  perform  die  task. 
The  time  estimates  provided  to  you  are  30  seconds  for  the  fastest  time,  1  minute  for  the  normal 
time,  and  2  minutes  for  the  slowest  time.  You  think  about  the  particular  airman  you  are  rating 
and  decide  that  he  takes  a  little  longer  than  die  normal  time,  but  not  as  long  as  die  slow  time  and 
decide  to  give  him  1  minute  and  22  seconds  as  die  time  estimate.  Therefore,  you  would  put  a  "0" 
for  zero  hours  in  the  "Hr”  box,  a  "1"  for  1  minute  in  die  Min  box,  and  a  "22”  for  22  seconds  in 
the  "Sec"  box.  Are  there  any  questions? 

We  are  also  asking  you  to  give  an  overall  judgment  about  each  airman  regarding  die  relative 
amount  of  acceptable  work  they  can  perform  in  a  typical  day.  This  rating  scale  is  presented  on 
the  last  page  of  your  Time  Estimation  Form.  The  question  is  worded: 

’In  your  specialty,  consider  the  maximum  amount  of  acceptable  work  that  can  be  done 
by  a  person  in  a  typical  day  as  100  percent  What  percent  of  the  maximum  could  die 
person  you  are  currently  rating  do  in  a  typical  day?  Write  your  estimate  in  die  box 
below.’ 

Look  at  the  example  below.  Assume  that  100%  is  the  maximum  amount  of  work  that  can  be 
done  and  1%  is  the  least  amount  of  work.  If  you  believe  that  the  airman  you  are  rating  can 
perform  75%  of  the  maximum  amount  of  work,  write  75  in  die  box  provided." 


95 


In  making  your  estimates,  please  think  about  how  long  it  would  take  each  airman  to  do  each 

task  if  he  or  she  were  working  as  quickly  as  they  could,  while  maintaining  satisfactory 

performance.  The  key  words  here  are  working  quickly  and  satisfactory  performance. 

Supervisor  Training 

Before  beginning  work  on  estimating  your  times,  I*d  like  to  go  over  some  basic  information  on 
making  these  estimates.  First,  we  need  to  establish  what  we  mean  by  performance  time.  In  making  your 
estimates,  please  think  about  how  long  it  would  take  each  airman  to  do  each  task  if  he  or  she  were 
working  as  quickly  as  they  could,  while  maintaining  satisfactory  performance.  The  key  words  here  are 
working  quickly  and  satisfactory  performance. 

There  are  some  other  things  we  need  to  clarify  before  we  go  on.  In  making  your  estimates,  we 
want  you  to  carefully  consider  each  airman  and  each  task  separately,  hi  making  ratings  or  estimates  like 
this  we  all  have  a  tendency  to  generalize.  In  other  words,  if  we  think  that  a  particular  airman  is  generally 
a  pretty  good  performer,  the  tendency  is  to  give  him  or  her  fast  performance  times  on  ah  tasks  when,  in 
fact,  it  is  very  unlikely  that  he  or  she  is  truly  good  at  everything.  Your  estimates  should  reflect  the  true 
strengths  and  weaknesses  of  each  airman  on  the  different  tasks. 

We  also  want  you  to  evaluate  each  airman  in  the  context  of  their  maximum  level  of  performance, 
not  just  the  most  recent  performance  you  have  observed.  If  drey  had  a  bad  day  last  Friday  and  really 
messed  something  up,  don’t  make  that  your  baseline  for  evaluation.  Try  to  think  back  to  all  the  times  you 
have  seen  him  perform  the  various  tasks  and  make  your  estimate  on  the  basis  of  his  best  days. 

Finally,  we  want  you  to  only  consider  the  performance  of  your  airmen  on  the  specific  tasks  listed 
in  your  booklets.  If  they  are  extremely  strong  or  weak  in  other  areas  of  performance,  such  as  appearance, 
punctuality,  attitude,  etc.,  please  try  not  to  let  that  influence  your  estimates  of  their  performance  times  on 
these  specific  tasks. 

Are  there  any  more  questions  before  we  take  a  short  break?  [After  all  questions  are  answered, 
dismiss  the  group  for  10>15  minutes.  Remind  them  where  the  facilities  are.  Tell  them  to  return  to 
their  same  seat  and  leave  all  their  materials  where  they  are  seated.] 
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Final  Instructions 


Now  for  some  final  instructions  before  we  get  down  to  business.  After  you  complete  the  Time 
Estimation  Form  for  one  of  your  airmen,  please  hold  die  form  up  for  collection  and  we  will  give  you  a 
Time  Estimation  Form  for  the  next  airman  on  your  list  When  you  have  completed  the  last  airman  on 
your  list  turn  in  all  remaining  materials— the  Task  Booklet  your  Background  Information  Form  and  die 
Supervisor-Airman  List  At  that  point  you  are  free  to  go.  Take  as  much  time  as  you  need-this  is  not  a 
test  and  there  are  no  time  limits. 

Remember,  in  making  your  estimates,  please  think  about  how  long  it  would  tairp.  each  airman  to 
do  each  task  if  he  or  sire  were  working  as  quickly  as  they  could,  while  maintaining  satisfactory 
performance.  The  key  words  here  are  working  Quickly  and  satisfactory  performance. 

Again,  we  thank  you  for  your  time  and  participation  in  this  important  project  Unless  you  have 
any  other  questions,  you  may  begin. 
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AIRMAN  TESTING  SESSIONS 


Introduction 

Good  moming/aftemoon,  my  name  is _ and  I’m  with  the  Human  Resources 

Research  Organization.  We  are  a  contractor  hired  by  die  Air  Force  to  examine  better  ways  of  selecting 
new  airmen  and  placing  them  in  appropriate  jobs.  Since  we  are  not  experts  in  the  work  you  do,  your 
participation  in  this  study  is  extremely  important  in  helping  us  improve  how  the  Air  Force  manages  its 
people.  We  have  selected  four  Air  Force  specialties  for  data  collection  and  analysis:  122X0  (Aircrew  Life 
Support);  454X1  (AGE);  455X2  (Avionic  Communication  and  Navigation  Systems);  and  732X0 
(Personnel).  Everyone  here  should  be  in  one  of  these  four  specialties. 

We  have  handed  out  a  lot  of  paperwork  that  is  unique  to  your  specialty  so  we  have  divided  you 
up  into  four  groups.  [Identify  the  locations  of  the  four  groups  and  ensure  that  everyone  is  in  the 
right  section.] 

Basically,  what  we  will  be  asking  you  to  do  this  moming/aftemoon  is  to  give  us  your  opinions 
on  a  variety  of  subjects,  including  die  kinds  of  work  you  like  to  do  and  your  work  habits  and  attinwtet 
We  will  also  be  testing  some  of  you  to  determine  how  much  you  know  about  some  of  the  tasks  you  do 
in  your  jobs.  The  information  we  are  collecting  is  for  research  only;  none  of  it  will  appear  anywhere  in 
your  permanent  records  nor  will  it  be  used  in  any  way  to  affect  your  careers.  Later  this  week,  we  will 
be  visiting  your  duty  sections  to  observe  some  of  you  at  work. 

Before  we  go  on  are  there  any  questions? 

Before  we  start,  we’d  like  to  go  through  die  list  of  names  of  airmen  who  should  be  here  today. 
Please  say  "here''  when  your  name  is  called.  [Take  roll.]  Is  there  anyone  here  whose  name  I  did  not 
call?  [Pass  out  a  3x5  card  and  ask  them  to  record  their  name,  rank,  AFSC,  unit,  and  first-line 
supervisor.] 

Background  Information  Forms  and  Privacy  Act  Statements 
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We’d  like  to  begin  by  having  you  fill  out  a  Background  Information  Form  with  a  Privacy  Act 
Statement  on  it  Please  read  the  Privacy  Act  Statement  and  take  a  few  minutes  to  fill  out  the  form.  The 
form  asks  for  some  basic  background  information  on  you  and  is  pretty  straightforward.  All  the  forms  you 
will  fill  out  today  must  be  done  with  a  Number  2  pencil. 

Now  look  at  the  standard  multiple-choice  scannable  answer  sheet  This  morning  you  will  be  given 
three  different  questionnaires,  but  will  be  using  only  this  one  answer  sheet  to  record  all  of  your  answers. 
As  with  all  answer  sheets  like  this,  be  sure  to  fill  in  only  one  circle  for  each  item.  Fill  in  the  circle 
completely,  erase  any  stray  marks,  and  if  you  have  to  change  an  answer,  erase  your  first  mark  completely 
and  then  mark  your  new  choice.  Please  notice  that  the  form  is  in  3  parts- -part  1  is  on  the  front  and  parts 
2  and  3  are  on  the  back. 

The  front  side  also  has  space  for  your  last  name,  your  Social  Security  number  and  your  Air  Force 
Specialty  Code.  Please  go  to  the  last  box  in  the  name  section.  We  are  using  this  box  to  record  the  base 

you  are  assigned  to.  Since  you  are  all  assigned  to _ Air  Force  base,  we  need  all  of  you  to  put 

the  letter _ in  the  last  box  and  fill  in  the  circle  for  this  letter  down  below.  Now  go  ahead  and  print  your 

last  name  in  the  blocks  in  the  "Last  Name"  section  and  fill  in  the  appropriate  circles.  Now  go  down  and 
do  the  same  thing  for  your  Social  Security  number  and  your  AFSC,  first  writing  the  numbers  in  the  boxes 
and  then  filling  in  die  appropriate  circle  below  each  number.  Please  code  in  your  skill  level  as  the  fourth 
digit  of  your  AFSC.  [Use  duty  AFSC  with  current  skill  level.]  Does  anyone  have  any  questions  so  far? 

Vocational  Interest  for  Career  Enhancement 

The  first  questionnaire  we  have  for  you  today  is  the  Vocational  Interest  for  Career  Enhancement, 
or  VOICE.  Please  don’t  start  yet  and  please  do  not  mark  in  these  booklets  since  we  have  to  use  diem 
again  at  other  bases.  Some  of  you  may  have  taken  this  when  you  went  through  basic  training  at  Lackland 
Air  Force  Base,  but  we  need  for  you  to  do  it  again,  since  your  interests  may  have  changed  since  them 
These  questions  are  designed  to  tell  us  die  kind  of  work  you  like  to  do.  This  is  not  a  test,  so  there  are 
no  right  or  wrong  answers.  Each  item  consists  of  a  description  of  some  activity  or  job.  For  each  activity 
you  are  to  indicate  on  die  answer  sheet  whether  you  like,  dislike,  or  are  indifferent  about  (neither  like  nor 
dislike)  the  activity.  Please  carefully  fill  in  the  circle  marked  "L"  for  like,  T  for  indifferent,  or  "D"  for 
dislike  next  to  each  item  number  on  your  answer  sheet,  again  using  only  a  number  2  pencil. 
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Please  open  the  VOICE  booklet  to  the  first  page  and  read  along  silently  as  I  read  aloud. 


"The  purpose  of  this  inventory  is  to  determine  which  of  a  number  of  occupations  you  would  like. 

This  is  not  an  intelligence  test  or  a  test  of  special  abilities.  There  are  no  right  answers.  The  right 

answer  for  you  is  the  one  that  best  describes  your  liking  for  the  type  of  work  or  activity  presented. 

All  your  marks  should  be  made  on  the  answer  sheet  provided  with  this  booklet  Make  sure  you 

use  the  soft  lead  pencil  provided  or  any  other  soft  lead  pencil.  Do  not  mark  on  the  booklet" 

And  now  we  will  go  on  to  the  instructions  for  completing  the  VOICE. 

There  are  four  sections  to  VOICE  and  part  1  of  your  answer  sheet  is  divided  into  four  sections 
corresponding  to  tire  sections  in  die  booklet  Section  1  is  a  set  of  job  titles;  section  2  is  a  list  of  duties 
from  a  number  of  different  jobs;  section  3  lists  leisure  time  activities;  and  section  4  lists  things  you  might 
want  to  study.  Read  the  instructions  at  the  beginning  of  each  section  and  fill  in  one  circle  for  each  item 
on  your  answer  sheet  Remember  "L"  means  you  like  the  activity,  "I"  means  you  neither  like  nor  dislike. 
the  activity,  and  "D"  means  you  dislike  die  activity. 

There  is  no  time  limit,  but  work  fairly  quickly  and  give  your  first  reaction  to  each  item.  Work 
through  all  four  sections  of  the  VOICE  before  stopping  and  remain  seated  until  everyone  is  finished.  If 
there  aren’t  any  questions,  you  may  begin.  [Pause.  Should  take  about  30  minutes.] 

It  looks  like  everyone  is  finished.  Please  keep  your  answer  sheets  but  pass  the  VOICE  booklets 
forward  so  we  can  collect  them.  [Collect  all  VOICE  booklets.] 

General  Scale 

Next  we  want  to  get  some  of  your  opinions  and  attitudes  toward  work  in  general.  The  next 
questionnaire  in  your  packet  is  called  the  General  Scale.  Please  don’t  start  yet  and,  again,  please  do  not 
mark  in  the  booklets.  This  booklet  has  only  30  items  which  correspond  to  Part  2  on  the  back  of  your 
answer  sheet  You  will  be  asked  to  agree  or  disagree  with  a  series  of  statements.  Look  at  tire  instructions 
printed  on  the  front  page  of  the  General  Scale  booklet  Please  read  along  silently  as  I  read  aloud. 

"To  familiarize  yourself  with  this  questionnaire,  it  would  be  useful  to  scan  it  quickly  before 

answering  any  of  tire  items.  It  is  a  good  idea  to  work  rather  fast  and  to  give  your  first  reaction 
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to  each  item.  Some  of  the  statements  may  seem  alike  to  you,  but  all  of  than  are  necessary  to 
show  slight  differences  of  opinion.  Because  your  opinion  is  being  asked  for,  there  are  no  right 
or  wrong  answers.  Also,  keep  in  mind  that  the  responses  to  this  questionnaire  are  being  kept 
confidential,  so  answer  items  as  honestly  as  you  can. 

Please  do  not  make  any  marks  in  this  booklet  You  should  record  your  answers  on  Part  2  • 
General  Scale,  located  on  Side  2  of  the  separate  answer  sheet  Mark  the  circle  on  your  answer 
sheet  that  corresponds  with  your  opinion  using  the  following  scale: 

Blacken  die  circle  marked  "A"  if  you  strongly  disagree  with  the  statement 

Blacken  the  circle  marked  "B"  if  you  disagree  with  the  statement 

"C  if  you  slightly  disagree 

"D"  if  you  neither  agree  nor  disagree 

"E"  if  you  slightly  agree 

"F  if  you  agree 

"G"  if  you  strongly  agree 

The  scale  is  printed  at  the  top  of  die  following  two  pages  for  reference.  When  you  are  told  to 
begin,  read  through  the  questionnaire  items  one  at  a  time  and  begin  recording  your  answers  on 
the  "General  Scale"  section  on  the  answer  sheet" 

When  everyone  is  finished  we  will  give  you  a  break,  but  please  remain  seated  until  all  are  done. 
You  may  begin.  (Pause  while  all  complete.] 

Please  pass  the  General  Scale  booklets  forward  so  we  can  collect  them.  [Collect  General  Scale 
Booklets.]  Those  of  you  in  the  455X2  (Avionic  Communication  and  Navigation  Systems)  career  field 
are  now  finished.  [Point  to  these  people.]  Please  pass  your  Background  Information  Forms,  answer 


sheets,  and  pencils  forward  to  be  collected.  [Collect  forms,  answer  sheets,  and  pencils.]  We  really 
appreciate  your  help  and  will  be  seeing  some  of  you  later  on  this  week  in  your  duty  sections.  [Wait  for 
455X2  airmen  to  leave  the  room.]  For  those  of  you  in  Aircrew  Life  Support,  Personnel,  and  AGE,  we 
have  one  more  thing  for  you  to  do,  but  first  we’ll  take  a  15  minute  break.  Please  leave  your  answer  sheet 
on  the  top  of  your  desk/table  and  return  to  your  same  seat  after  the  break.  [Remind  them  where  the 
facilities  are.] 

Job  Knowledge  Tests 

For  those  of  you  remaining  we  have  just  one  more  task.  This  is  the  Job  Knowledge  Test  for  your 
career  field.  Again,  we  ask  that  you  not  mark  in  these  books  so  we  can  use  them  again.  Unlike  the  first 
two  questionnaires  we  did,  die  questions  on  the  Job  Knowledge  Test  have  one  correct  answer,  so  you 
should  do  your  best  to  pick  the  correct  answer.  You  will  record  the  answers  to  these  questions  in  Part 
3  on  the  back  of  your  answer  sheets  in  the  area  labeled  "Part  3  -  Job  Knowledge  Test,"  again  using  only 
a  number  2  pencil.  Does  anyone  need  another  pencil? 

All  of  the  questions  arc  multiple  choice,  having  up  to  5  choices.  After  you  read  the  question  or 
statement  and  decide  on  the  correct  response— there  is  only  one  best  response  for  each  item-fill  in  the 
circle  for  the  letter  corresponding  to  your  choice  on  the  answer  sheet  Open  your  Job  Knowledge  Tests 
to  page  1  and  read  along  as  I  read  aloud: 

"Each  item  in  this  booklet  consists  of  a  question  or  statement  followed  by  several  choices.  There 
is  only  one  choice  that  answers  the  question  or  completes  the  statement  correctly.  Be  sure  to  read 
each  question  and  all  of  the  choices  before  answering.  Decide  which  choice  is  correct  and 
blacken  the  letter  on  your  answer  sheet  that  matches  the  letter  and  item  number.” 

The  example  in  your  instructions  shows  "C  filled  in  since  Washington,  DC  is  the  capital  of  the 
United  States  and  "C"  is  the  correct  answer. 

"Be  sure  to  use  a  number  2  pencil  and  blacken  only  one  circle  for  each  item.  Erase  any  stray 
marks  being  careful  not  to  tear  the  answer  sheet  If  you  have  to  change  an  answer,  erase  your 
first  mark  completely,  and  then  mark  your  new  choice. 
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The  questions  in  this  booklet  are  grouped  by  task.  Read  the  task  number  and  title  at  the  top  of 
each  page  to  detennine  when  you  have  started  a  new  task.  [Point  to  task  number  and  title.] 

Do  not  spend  too  much  time  on  any  one  item.  If  you  have  trouble  with  an  item,  skip  it,  and  come 
back  to  it  after  you  finish  the  other  items.  Although  you  may  be  unfamiliar  with  a  task,  make  the 
best  choice  you  can  for  each  item.  Try  to  answer  every  item." 

The  three  tests  are  of  different  lengths,  so  don’t  be  concerned  if  you  don’t  use  all  of  the  items  on 
your  answer  sheet  or  if  some  people  in  the  room  finish  before  you.  This  is  not  a  timed  test,  so  take  as 
much  time  as  you  need.  Are  there  any  questions  before  we  begin?  [Pause.] 

When  you  are  finished,  please  bring  the  test  book,  your  answer  sheet,  and  your  Background 
Information  Form  to  the  front  of  the  room  before  you  leave.  We  would  also  like  to  have  the  pencils  back. 
We  thank  you  for  your  help  in  this  important  project  If  there  are  no  further  questions,  you  may  begin. 
[Collect  materials  as  airmen  finish.] 
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TIMING  SESSIONS 


Instructions  to  Senior  NCOs 


Introduction 

We  arc  researchers  from  the  Air  Force  Armstrong  Laboratory,  Human  Resources  Directorate 
(formerly  the  Air  Force  Human  Resources  Laboratory)  and  (me  of  its  contractors,  the  Human  Resources 
Research  Organization  (HumRRO).  We  are  here  this  week  collecting  data  for  a  research  project 
examining  better  ways  of  selecting  and  classifying  new  recruits.  This  phase  of  the  project  will  consist  of 
timing  a  few  airmen  in  your  career  field  as  they  perform  some  common  tasks. 

Task  Booklets 

Because  we  know  very  little  about  foe  work  done  in  this  specialty,  we  have  asked  for  your  assistance 
in  setting  up  foe  tasks  to  be  timed  and  in  evaluating  foe  quality  of  each  airman’s  performance.  The  tasks 
we  are  going  to  time  are  described  in  this  booklet  [provide  a  copy  of  the  appropriate  task  booklet]  As 
you  will  notice,  each  task  is  numbered  and  starts  with  a  one  or  two  sentence  description  of  the  work  to 
be  performed  followed  by  a  set  of  force  times  referred  to  as  "Time  Estimates".  These  times  were 
estimated  recently  by  a  group  of  sue  NCOs  from  your  career  field  in  a  workshop  at  Brooks  AFB. 

We  will  not  be  using  any  of  these  times  today  except  to  establish  a  cut-off  for  those  airman  who  are 
having  extreme  difficulty  completing  a  task.  We  will  say  "STOP"  if  the  airman  reaches  the  cutoff  time. 

The  task  descriptions  in  the  booklet  also  include  a  list  of  the  equipment  and  materials  needed  to 
perform  each  task-these  lists  were  provided  to  you  prior  to  our  visit  Have  you  had  any  problem 
gathering  up  die  materials? 

The  task  descriptions  also  include  a  little  background  on  what  we  are  attempting  to  measure  with  the 
task  and  a  description  of  the  configuration  of  the  equipment  and  environment  at  die  time  the  task  is  being 
performed.  We  want  to  try  and  keep  this  configuration  die  same  for  each  airman  who  performs  die  task, 
so  the  resulting  times  are  comparable.  The  task  booklet  also  contains  a  set  of  narrative  instructions  to  be 
verbally  given  to  foe  airman  and  a  list  of  the  steps  and  actions  the  airman  should  take  to  properly  perform 
foe  task  Let’s  look  over  a  few  tasks  in  the  booklet  to  see  if  you  have  any  questions,  [review  tasks  with 
the  NCO] 


Timing  Forms 

Now  let’s  go  over  the  data  sheet  we  will  be  using  to  record  information  cm  each  airman  as  they 
perform  some  of  these  tasks.  At  foe  top  of  foe  form  we  have  recorded  die  airman’s  name,  SSAN,  grade 
AFS  and  base  of  assignment  so  we  can  later  match  their  task  performance  times  with  other  data  we  are 
collecting.  The  form  then  includes  space  to  record  information  on  up  to  eleven  tasks.  For  each  task  we 
have  recorded  the  task  number  and  tide  from  the  booklet,  and  the  cut-off  time  for  that  task.  During  the 
timing,  we  will  be  recording  the  actual  time  it  took  that  airman  to  do  the  task,  and  finally  a  proficiency 
rating,  to  be  provided  by  you,  indicating  how  well  die  airman  did  the  task.  This  rating  is  on  a  1  to  5  scale 
with  5  being  the  best  and  1  being  die  worst  Please  review  die  scale  description  printed  on  the  green  card 
[hand  out  rating  card]. 

5  ■  Exceptional  Far  exceeds  foe  acceptable  level  of  performance.  Completed  all  steps,  critical  and 
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noncritical,  efficiently  (skillfully)  and  accurately. 


4  =  Distinguished  Exceeds  the  acceptable  level  of  performance.  Completed  all  critical  steps 

accurately,  but  not  necessarily  efficiently.  Made  very  few  or  no  noncritical  errors. 

3  =  Acceptable  Meets  the  minimum  acceptable  level  of  performance.  Completed  all  critical  steps 

accurately.  Made  some  noncritical  errors. 

2  =  Deficient  Below  the  acceptable  level  of  performance.  Made  some  critical  errors  or 

numerous  noncritical  errors. 

1  =  Unacceptable  Far  below  the  acceptable  level  of  performance.  Made  numerous  critical  and 
noncritical  errors  and/or  could  not  complete  the  ta.sk 

Critical  steps  are  those  steps  that  absolutely  must  be  performed  to  correctly  arid  safely  complete  the 
task.  Noncritical  steps  are  steps  of  less  importance;  failure  to  perform  a  noncritical  step  will  not 
necessarily  lead  to  incorrect  overall  performance  of  the  task,  or  an  unsafe  condition.  Do  you  have  any 
questions  about  the  data  collection  forms  or  the  rating  scale?  [respond  to  questions] 

Timing  Procedure 

The  first  step  in  collecting  this  data  will  be  to  set  up  all  the  materials  required  for  the  task.  Again, 
to  help  ensure  consistency  we  want  to  be  sure  that  the  set-up  is  die  same  for  each  airman  that  we  time. 

The  next  step  will  be  to  instruct  the  airman.  Before  we  have  you  give  the  airman  the  specific 
instructions  for  the  task  to  be  performed,  we  will  give  him  (or  her)  some  general  ground  rules.  We  will 
tell  the  airman  that  we  want  him  (or  her)  to  work  as  quickly  as  possible,  while  completing  die  task  to  a 
satisfactory  level  of  performance.  We  want  to  find  out  how  quickly  the  airman  can  perform  the  task  to 
a  satisfactory  level.  We  will  also  assure  him  (or  her)  that  the  data  is  being  collected  for  research  purposes 
only  and  will  have  no  impact  on  him  (or  her)  personally. 

We  will  also  instruct  the  airman  that  after  we  tell  them  to  "begin”  (and  start  the  dock)  he  should 
keep  working  at  the  task  until  they  either  complete  it  or  we  tell  him  to  stop.  When  he  thinks  he  is 
finished,  he  is  to  say  "finished"  and  we  will  stop  the  clock  and  record  his  performance  time.  He  will  be 
told  that  he  may  not  ask  any  questions  or  request  assistance  once  die  timing  begins.  You  may  use  the  task 
step  descriptions  in  the  booklet  to  keep  track  of  how  many  steps  are  correctly  performed  to  help  you  make 
the  proficiency  rating  when  the  task  is  complete.  If  you  see  an  airman  doing  something  wrong,  do  not 
say  anything  or  help  him.  If  an  airman  is  totally  unable  to  accomplish  a  task,  we  will  let  him  work  at  it 
until  die  cut-off  time  is  reached  and  then  tell  them  to  stop.  If  this  occurs,  we  will  record  the  cut-off  time 
as  his  task  performance  time. 

If  an  airman  is  not  qualified  to  perform  a  given  task,  we  would  still  like  die  airman  to  attempt  the 
task.  This  is  provided  that  it  would  be  safe  for  him  or  her  to  do  so  and  that  it  would  not  violate  any 
regulations.  (If  an  airman  refuses  to  do  a  task,  he  or  she  should  be  encouraged  to  do  it,  but  not  forced.) 

If  the  airman  does  not  finish  the  task  by  the  cutoff  time,  he  will  be  given  a  rating  of  "1"  which  is 
unacceptable.  If  the  airman  states  that  he  has  completed  the  task,  you  may  give  him  any  rating. 

After  an  airman  finishes  one  task  we  will  immediately  begin  the  set-up  for  die  next  task.  We  have 
a  lot  of  timing  to  do  this  week,  so  we  must  keep  die  airmen  moving  through  the  tasks  as  quickly  as 
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possible.  When  one  airman  has  gone  through  all  the  tasks,  we  will  then  start  over  with  the  next  airman. 


After  we  give  the  airman  the  general  instructions,  we  will  let  you  read  them  the  specific  instructions 
for  the  task  and  ask  if  they  have  any  questions.  [Show  NCO  the  part  that  they  are  to  read.]  You  can 
answer  questions  relating  to  the  timing  procedures,  but  do  not  give  them  any  help  in  how  to  actually 
perform  the  task  itself-we  are  trying  to  measure  their  proficiency,  not  yours.  Do  you  have  any  questions 
or  comments  before  we  get  started?  [respond  to  questions] 


114 


TIMING  SESSIONS 


Instructions  to  airmen  being  timed 


Introduction 

We  are  researchers  from  the  Air  Force  Ann  strong  Laboratory,  Human  Resources  Directorate 
(formerly  die  Air  Force  Human  Resources  Laboratory)  and  one  of  its  contractors,  the  Human  Resources 
Research  Organization  (HumRRO).  We  are  here  this  week  collecting  data  for  a  research  project 
examining  better  ways  of  selecting  and  classifying  new  recruits.  This  phase  of  the  project  will  consist  of 
timing  you  and  a  few  other  airmen  in  your  career  field  as  you  perform  some  common  tasks.  We  really 
appreciate  your  participation  in  this  project  The  data  we  collert  will  help  die  Air  Force  do  a  better  job 
of  picking  die  right  people  and  placing  them  in  the  proper  career  fields.  Earlier  this  week  we  got  some 
written  information  from  you  on  your  job  interests  and  attitudes  and  now  we  need  to  get  some  actual  job 
performance  data.  We  will  be  timing  you  on  &  number  of  tasks  performed  by  airmen  in  your  career  field. 

As  we  said  earlier  this  week,  die  information  we  are  collecting  is  for  research  only.  Nothing  we 
collect  regarding  your  performance  will  become  part  of  your  records  nor  will  it  be  used  in  any  way  to 
evaluate  you  or  affect  your  career. 

Timing  Procedures 

Sgt _ here  will  be  helping  us  set  up  a  few  tasks  commonly  done  by  airmen  and  junior 

NCOs  in  your  career  field.  He  will  give  you  some  instructions  on  what  we  want  done  and  then  will  watch 
as  you  perform  the  task  while  we  time  you.  Before  he  gives  you  die  instructions  for  the  first  task  we  need 
to  go  over  a  few  things. 

First,  please  look  over  die  information  at  die  top  of  this  Timing  Form  to  be  sure  we  have  your  name, 
SSAN,  grade  and  AFS  correct  Let  us  know  if  there  are  any  errors,  [mark  any  corrections  required] 

Before  we  start  timing,  you  may  ask  questions  about  the  timing  procedures;  however,  you  may  not 
ask  for  help  in  how  to  actually  perform  die  task-that  is  what  we  are  trying  to  measure.  Once  we  say 
"begin"  and  die  timing  starts  you  should  work  as  quickly  as  possible,  without  any  assistance,  until  you 
feel  you  have  satisfactorily  completed  the  task,  at  which  time  you  are  to  say  "finished".  However,  if 
during  the  task  you  would  like  to  have  the  instructions  repeated,  then  you  may  ask.  We  want  to  see  how 
fast  you  can  complete  the  task  while  doing  it  satisfactorily.  If  you  are  having  trouble  completing  a  task, 
don’t  panic  or  worry  about  it-just  keep  trying  until  we  tell  you  to  stop. 

After  you  finish  one  task  we  will  immediately  set  up  the  next  one  so  we  can  get  through  all  the 
airmen  and  tasks  we  have  to  time  today .  If  you  need  a  break  between  tasks,  let  us  know-but  please  don’t 
take  a  break  during  any  of  the  tasks.  Do  you  have  any  questions?  [respond  to  questions] 

Sgt _ will  now  give  you  instructions  for  the  first  task.  Remember,  do  not  start  until  we 

say  "begin”,  no  asking  for  help  once  you  begin,  and  say  "finished"  when  you  are  done.  Remember  to 
perform  die  task  as  quickly  as  you  can,  while  performing  at  a  satisfactory  level  of  performance.  Over  to 
you  Sgt _ . 
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